Oct 30 10:44:29.464705 ip-10-0-137-242 systemd[1]: Starting Kubernetes Kubelet... Oct 30 10:44:29.839994 ip-10-0-137-242 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 10:44:29.839994 ip-10-0-137-242 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 10:44:29.839994 ip-10-0-137-242 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 10:44:29.839994 ip-10-0-137-242 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 10:44:29.845476 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845375 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 10:44:29.845476 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845472 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 10:44:29.845476 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845478 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 10:44:29.845476 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845482 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 10:44:29.845717 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845489 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 10:44:29.845717 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845496 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 10:44:29.845717 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845501 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 10:44:29.845717 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845505 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 10:44:29.845717 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845508 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 10:44:29.845717 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845512 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 10:44:29.845717 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845516 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 10:44:29.845717 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845520 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 10:44:29.845717 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845524 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 10:44:29.845717 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845528 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 10:44:29.845717 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845532 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 10:44:29.845717 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845538 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 10:44:29.845717 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845544 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 10:44:29.845717 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845548 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 10:44:29.845717 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845552 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 10:44:29.845717 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845558 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 10:44:29.845717 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845562 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 10:44:29.845717 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845567 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 10:44:29.845717 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845571 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 10:44:29.846553 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845575 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 10:44:29.846553 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845579 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 10:44:29.846553 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845583 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 10:44:29.846553 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845588 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 10:44:29.846553 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845592 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 10:44:29.846553 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845597 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 10:44:29.846553 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845601 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 10:44:29.846553 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845605 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 10:44:29.846553 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845610 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 10:44:29.846553 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845615 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 10:44:29.846553 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845619 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 10:44:29.846553 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845623 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 10:44:29.846553 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845628 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 10:44:29.846553 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845632 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 10:44:29.846553 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845638 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 10:44:29.846553 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845643 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 10:44:29.846553 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845647 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 10:44:29.846553 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845652 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 10:44:29.846553 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845656 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 10:44:29.846553 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845660 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 10:44:29.847364 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845664 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 10:44:29.847364 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845669 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 10:44:29.847364 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845673 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 10:44:29.847364 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845677 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 10:44:29.847364 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845681 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 10:44:29.847364 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845685 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 10:44:29.847364 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845690 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 10:44:29.847364 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845697 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 10:44:29.847364 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845702 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 10:44:29.847364 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845706 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 10:44:29.847364 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845711 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 10:44:29.847364 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845715 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 10:44:29.847364 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845719 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 10:44:29.847364 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845723 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 10:44:29.847364 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845727 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 10:44:29.847364 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845731 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 10:44:29.847364 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845736 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 10:44:29.847364 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845740 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 10:44:29.847364 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845744 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 10:44:29.848207 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845749 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 10:44:29.848207 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845754 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 10:44:29.848207 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845758 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 10:44:29.848207 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845762 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 10:44:29.848207 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845766 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 10:44:29.848207 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845771 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 10:44:29.848207 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845775 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 10:44:29.848207 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845781 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 10:44:29.848207 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845785 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 10:44:29.848207 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845789 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 10:44:29.848207 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845794 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 10:44:29.848207 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845799 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 10:44:29.848207 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845803 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 10:44:29.848207 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845808 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 10:44:29.848207 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845812 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 10:44:29.848207 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845816 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 10:44:29.848207 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845820 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 10:44:29.848207 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845824 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 10:44:29.848207 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845829 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 10:44:29.848207 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845833 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 10:44:29.849059 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845837 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 10:44:29.849059 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845843 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 10:44:29.849059 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845848 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 10:44:29.849059 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845852 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 10:44:29.849059 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845856 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 10:44:29.849059 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845860 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 10:44:29.849059 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845865 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 10:44:29.849059 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845869 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 10:44:29.849059 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845873 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 10:44:29.849059 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845877 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 10:44:29.849059 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845881 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 10:44:29.849059 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845886 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 10:44:29.849059 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845890 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 10:44:29.849059 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845895 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 10:44:29.849059 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845899 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 10:44:29.849059 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845903 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 10:44:29.849059 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845907 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 10:44:29.849059 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845912 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 10:44:29.849059 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845916 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 10:44:29.849059 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845923 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 10:44:29.849636 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845927 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 10:44:29.849636 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845932 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 10:44:29.849636 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845936 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 10:44:29.849636 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.845941 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 10:44:29.849636 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847242 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 10:44:29.849636 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847254 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 10:44:29.849636 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847259 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 10:44:29.849636 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847264 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 10:44:29.849636 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847271 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 10:44:29.849636 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847277 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 10:44:29.849636 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847281 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 10:44:29.849636 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847286 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 10:44:29.849636 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847291 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 10:44:29.849636 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847296 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 10:44:29.849636 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847301 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 10:44:29.849636 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847306 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 10:44:29.849636 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847311 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 10:44:29.849636 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847315 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 10:44:29.849636 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847319 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 10:44:29.850226 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847324 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 10:44:29.850226 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847328 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 10:44:29.850226 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847332 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 10:44:29.850226 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847336 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 10:44:29.850226 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847341 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 10:44:29.850226 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847345 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 10:44:29.850226 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847349 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 10:44:29.850226 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847353 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 10:44:29.850226 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847357 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 10:44:29.850226 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847361 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 10:44:29.850226 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847365 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 10:44:29.850226 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847369 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 10:44:29.850226 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847373 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 10:44:29.850226 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847377 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 10:44:29.850226 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847382 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 10:44:29.850226 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847386 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 10:44:29.850226 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847390 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 10:44:29.850226 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847395 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 10:44:29.850226 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847399 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 10:44:29.851066 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847403 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 10:44:29.851066 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847408 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 10:44:29.851066 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847412 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 10:44:29.851066 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847416 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 10:44:29.851066 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847420 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 10:44:29.851066 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847425 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 10:44:29.851066 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847444 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 10:44:29.851066 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847451 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 10:44:29.851066 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847455 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 10:44:29.851066 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847468 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 10:44:29.851066 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847475 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 10:44:29.851066 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847479 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 10:44:29.851066 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847484 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 10:44:29.851066 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847488 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 10:44:29.851066 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847492 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 10:44:29.851066 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847498 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 10:44:29.851066 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847502 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 10:44:29.851066 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847507 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 10:44:29.851066 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847511 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 10:44:29.851912 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847515 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 10:44:29.851912 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847519 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 10:44:29.851912 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847524 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 10:44:29.851912 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847528 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 10:44:29.851912 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847532 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 10:44:29.851912 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847536 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 10:44:29.851912 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847540 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 10:44:29.851912 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847545 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 10:44:29.851912 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847549 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 10:44:29.851912 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847553 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 10:44:29.851912 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847557 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 10:44:29.851912 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847562 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 10:44:29.851912 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847566 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 10:44:29.851912 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847570 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 10:44:29.851912 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847575 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 10:44:29.851912 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847579 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 10:44:29.851912 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847583 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 10:44:29.851912 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847587 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 10:44:29.851912 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847591 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 10:44:29.852587 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847596 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 10:44:29.852587 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847601 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 10:44:29.852587 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847606 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 10:44:29.852587 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847610 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 10:44:29.852587 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847617 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 10:44:29.852587 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847621 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 10:44:29.852587 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847625 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 10:44:29.852587 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847630 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 10:44:29.852587 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847634 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 10:44:29.852587 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847638 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 10:44:29.852587 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847642 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 10:44:29.852587 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847646 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 10:44:29.852587 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847650 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 10:44:29.852587 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847654 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 10:44:29.852587 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847659 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 10:44:29.852587 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847662 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 10:44:29.852587 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847667 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 10:44:29.852587 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847671 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 10:44:29.852587 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847675 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 10:44:29.852587 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847680 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 10:44:29.853263 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847684 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 10:44:29.853263 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847688 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 10:44:29.853263 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847692 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 10:44:29.853263 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847697 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 10:44:29.853263 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847701 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 10:44:29.853263 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847705 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 10:44:29.853263 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847709 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 10:44:29.853263 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847714 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 10:44:29.853263 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847718 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 10:44:29.853263 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847722 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 10:44:29.853263 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847727 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 10:44:29.853263 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847731 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 10:44:29.853263 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847735 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 10:44:29.853263 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.847742 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 10:44:29.853263 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848587 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 10:44:29.853263 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848601 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 10:44:29.853263 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848607 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 10:44:29.853263 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848612 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 10:44:29.853263 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848616 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 10:44:29.853773 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848621 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 10:44:29.853773 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848625 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 10:44:29.853773 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848629 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 10:44:29.853773 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848636 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 10:44:29.853773 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848642 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 10:44:29.853773 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848648 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 10:44:29.853773 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848653 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 10:44:29.853773 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848657 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 10:44:29.853773 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848662 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 10:44:29.853773 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848666 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 10:44:29.853773 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848671 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 10:44:29.853773 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848675 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 10:44:29.853773 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848679 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 10:44:29.853773 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848684 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 10:44:29.853773 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848688 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 10:44:29.853773 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848692 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 10:44:29.853773 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848697 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 10:44:29.853773 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848701 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 10:44:29.853773 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848705 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 10:44:29.854237 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848709 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 10:44:29.854237 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848714 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 10:44:29.854237 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848718 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 10:44:29.854237 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848722 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 10:44:29.854237 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848726 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 10:44:29.854237 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848730 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 10:44:29.854237 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848735 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 10:44:29.854237 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848741 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 10:44:29.854237 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848745 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 10:44:29.854237 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848753 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 10:44:29.854237 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848757 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 10:44:29.854237 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848762 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 10:44:29.854237 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848766 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 10:44:29.854237 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848770 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 10:44:29.854237 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848775 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 10:44:29.854237 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848779 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 10:44:29.854237 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848783 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 10:44:29.854237 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848787 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 10:44:29.854237 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848791 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 10:44:29.854237 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848795 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 10:44:29.854788 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848799 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 10:44:29.854788 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848804 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 10:44:29.854788 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848808 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 10:44:29.854788 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848812 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 10:44:29.854788 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848816 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 10:44:29.854788 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848821 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 10:44:29.854788 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848825 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 10:44:29.854788 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848829 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 10:44:29.854788 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848833 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 10:44:29.854788 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848838 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 10:44:29.854788 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848842 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 10:44:29.854788 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848846 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 10:44:29.854788 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848850 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 10:44:29.854788 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848854 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 10:44:29.854788 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848859 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 10:44:29.854788 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848863 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 10:44:29.854788 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848867 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 10:44:29.854788 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848871 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 10:44:29.854788 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848875 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 10:44:29.854788 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848881 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 10:44:29.855288 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848886 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 10:44:29.855288 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848891 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 10:44:29.855288 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848895 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 10:44:29.855288 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848900 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 10:44:29.855288 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848905 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 10:44:29.855288 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848910 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 10:44:29.855288 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848914 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 10:44:29.855288 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848918 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 10:44:29.855288 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848922 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 10:44:29.855288 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848926 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 10:44:29.855288 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848930 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 10:44:29.855288 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848934 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 10:44:29.855288 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848938 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 10:44:29.855288 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848943 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 10:44:29.855288 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848947 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 10:44:29.855288 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848951 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 10:44:29.855288 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848956 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 10:44:29.855288 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848960 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 10:44:29.855288 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848964 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 10:44:29.855771 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848968 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 10:44:29.855771 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848973 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 10:44:29.855771 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848977 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 10:44:29.855771 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848981 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 10:44:29.855771 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848986 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 10:44:29.855771 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848990 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 10:44:29.855771 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.848994 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 10:44:29.855771 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.849001 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 10:44:29.855771 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.849006 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 10:44:29.855771 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.849011 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 10:44:29.855771 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.849015 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 10:44:29.855771 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.849019 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 10:44:29.855771 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.849025 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 10:44:29.855771 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.849030 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 10:44:29.855771 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.849035 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 10:44:29.855771 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.849039 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 10:44:29.855771 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.849043 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 10:44:29.855771 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.849049 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 10:44:29.855771 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.849054 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 10:44:29.856212 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.849058 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 10:44:29.856212 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.849062 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 10:44:29.856212 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.849066 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 10:44:29.856212 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.849070 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 10:44:29.856212 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.849955 2579 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 10:44:29.856212 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.849971 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 10:44:29.856212 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.849983 2579 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 10:44:29.856212 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.849990 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 10:44:29.856212 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.849997 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 10:44:29.856212 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850002 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 10:44:29.856212 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850009 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 10:44:29.856212 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850017 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 10:44:29.856212 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850022 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 10:44:29.856212 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850027 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 10:44:29.856212 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850033 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 10:44:29.856212 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850038 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 10:44:29.856212 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850043 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 10:44:29.856212 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850048 2579 flags.go:64] FLAG: --cgroup-root="" Oct 30 10:44:29.856212 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850052 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 10:44:29.856212 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850057 2579 flags.go:64] FLAG: --client-ca-file="" Oct 30 10:44:29.856212 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850062 2579 flags.go:64] FLAG: --cloud-provider="external" Oct 30 10:44:29.856212 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850067 2579 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 10:44:29.856212 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850073 2579 flags.go:64] FLAG: --cluster-domain="" Oct 30 10:44:29.856758 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850077 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 10:44:29.856758 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850082 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 10:44:29.856758 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850088 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 10:44:29.856758 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850094 2579 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 10:44:29.856758 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850101 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 10:44:29.856758 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850107 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 10:44:29.856758 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850112 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 10:44:29.856758 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850117 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 10:44:29.856758 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850122 2579 flags.go:64] FLAG: --contention-profiling="false" Oct 30 10:44:29.856758 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850138 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 10:44:29.856758 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850143 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 10:44:29.856758 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850148 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 10:44:29.856758 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850153 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 10:44:29.856758 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850166 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 10:44:29.856758 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850171 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 10:44:29.856758 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850175 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 10:44:29.856758 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850180 2579 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 10:44:29.856758 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850185 2579 flags.go:64] FLAG: --enable-server="true" Oct 30 10:44:29.856758 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850189 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 10:44:29.856758 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850197 2579 flags.go:64] FLAG: --event-burst="100" Oct 30 10:44:29.856758 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850202 2579 flags.go:64] FLAG: --event-qps="50" Oct 30 10:44:29.856758 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850207 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 10:44:29.856758 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850211 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 10:44:29.856758 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850216 2579 flags.go:64] FLAG: --eviction-hard="" Oct 30 10:44:29.857423 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850223 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 10:44:29.857423 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850229 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 10:44:29.857423 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850234 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 10:44:29.857423 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850238 2579 flags.go:64] FLAG: --eviction-soft="" Oct 30 10:44:29.857423 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850243 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 10:44:29.857423 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850248 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 10:44:29.857423 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850253 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 10:44:29.857423 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850258 2579 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 10:44:29.857423 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850263 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 10:44:29.857423 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850269 2579 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 10:44:29.857423 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850273 2579 flags.go:64] FLAG: --feature-gates="" Oct 30 10:44:29.857423 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850280 2579 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 10:44:29.857423 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850285 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 10:44:29.857423 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850290 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 10:44:29.857423 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850295 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 10:44:29.857423 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850300 2579 flags.go:64] FLAG: --healthz-port="10248" Oct 30 10:44:29.857423 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850306 2579 flags.go:64] FLAG: --help="false" Oct 30 10:44:29.857423 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850310 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-137-242.ec2.internal" Oct 30 10:44:29.857423 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850316 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 10:44:29.857423 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850321 2579 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 10:44:29.857423 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850326 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 10:44:29.857423 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850332 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 10:44:29.857423 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850338 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850342 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850347 2579 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850352 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850357 2579 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850362 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850367 2579 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850372 2579 flags.go:64] FLAG: --kube-reserved="" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850376 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850381 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850387 2579 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850391 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850396 2579 flags.go:64] FLAG: --lock-file="" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850401 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850406 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850411 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850420 2579 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850425 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850444 2579 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850449 2579 flags.go:64] FLAG: --logging-format="text" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850453 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850459 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850463 2579 flags.go:64] FLAG: --manifest-url="" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850468 2579 flags.go:64] FLAG: --manifest-url-header="" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850475 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 10:44:29.857986 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850480 2579 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850486 2579 flags.go:64] FLAG: --max-pods="110" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850491 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850496 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850502 2579 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850506 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850511 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850516 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850520 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850532 2579 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850537 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850542 2579 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850547 2579 flags.go:64] FLAG: --pod-cidr="" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850552 2579 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850556 2579 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850561 2579 flags.go:64] FLAG: --pods-per-core="0" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850566 2579 flags.go:64] FLAG: --port="10250" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850571 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850576 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d0210ea6e5c5c0c5" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850581 2579 flags.go:64] FLAG: --qos-reserved="" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850586 2579 flags.go:64] FLAG: --read-only-port="10255" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850591 2579 flags.go:64] FLAG: --register-node="true" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850596 2579 flags.go:64] FLAG: --register-with-taints="" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850602 2579 flags.go:64] FLAG: --registry-burst="10" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850606 2579 flags.go:64] FLAG: --registry-qps="5" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850611 2579 flags.go:64] FLAG: --reserved-cpus="" Oct 30 10:44:29.858688 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850616 2579 flags.go:64] FLAG: --reserved-memory="" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850622 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850627 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850632 2579 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850637 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850641 2579 flags.go:64] FLAG: --runonce="false" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850646 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850651 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850656 2579 flags.go:64] FLAG: --seccomp-default="false" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850661 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850665 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850671 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850676 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850681 2579 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850686 2579 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850690 2579 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850698 2579 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850703 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850708 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850715 2579 flags.go:64] FLAG: --system-cgroups="" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850719 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850728 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850733 2579 flags.go:64] FLAG: --tls-cert-file="" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850737 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850744 2579 flags.go:64] FLAG: --tls-min-version="" Oct 30 10:44:29.859315 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850749 2579 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 10:44:29.859919 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850753 2579 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 10:44:29.859919 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850758 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 10:44:29.859919 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850763 2579 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 10:44:29.859919 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850768 2579 flags.go:64] FLAG: --v="2" Oct 30 10:44:29.859919 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850775 2579 flags.go:64] FLAG: --version="false" Oct 30 10:44:29.859919 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850781 2579 flags.go:64] FLAG: --vmodule="" Oct 30 10:44:29.859919 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850788 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 10:44:29.859919 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.850794 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 10:44:29.859919 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851030 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 10:44:29.859919 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851037 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 10:44:29.859919 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851042 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 10:44:29.859919 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851047 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 10:44:29.859919 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851051 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 10:44:29.859919 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851056 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 10:44:29.859919 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851060 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 10:44:29.859919 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851064 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 10:44:29.859919 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851068 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 10:44:29.859919 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851072 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 10:44:29.859919 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851077 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 10:44:29.859919 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851081 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 10:44:29.859919 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851085 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 10:44:29.860418 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851089 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 10:44:29.860418 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851097 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 10:44:29.860418 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851101 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 10:44:29.860418 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851106 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 10:44:29.860418 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851112 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 10:44:29.860418 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851116 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 10:44:29.860418 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851121 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 10:44:29.860418 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851125 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 10:44:29.860418 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851129 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 10:44:29.860418 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851133 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 10:44:29.860418 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851138 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 10:44:29.860418 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851142 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 10:44:29.860418 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851146 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 10:44:29.860418 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851151 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 10:44:29.860418 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851155 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 10:44:29.860418 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851160 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 10:44:29.860418 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851164 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 10:44:29.860418 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851170 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 10:44:29.860418 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851176 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 10:44:29.860888 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851182 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 10:44:29.860888 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851188 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 10:44:29.860888 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851192 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 10:44:29.860888 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851196 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 10:44:29.860888 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851200 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 10:44:29.860888 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851204 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 10:44:29.860888 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851209 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 10:44:29.860888 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851213 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 10:44:29.860888 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851217 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 10:44:29.860888 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851221 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 10:44:29.860888 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851226 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 10:44:29.860888 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851231 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 10:44:29.860888 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851235 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 10:44:29.860888 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851239 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 10:44:29.860888 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851246 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 10:44:29.860888 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851250 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 10:44:29.860888 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851255 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 10:44:29.860888 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851261 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 10:44:29.860888 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851265 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 10:44:29.860888 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851269 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 10:44:29.861363 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851274 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 10:44:29.861363 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851278 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 10:44:29.861363 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851282 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 10:44:29.861363 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851287 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 10:44:29.861363 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851291 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 10:44:29.861363 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851295 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 10:44:29.861363 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851299 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 10:44:29.861363 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851304 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 10:44:29.861363 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851308 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 10:44:29.861363 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851312 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 10:44:29.861363 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851317 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 10:44:29.861363 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851321 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 10:44:29.861363 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851325 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 10:44:29.861363 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851330 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 10:44:29.861363 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851334 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 10:44:29.861363 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851338 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 10:44:29.861363 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851342 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 10:44:29.861363 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851346 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 10:44:29.861363 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851351 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 10:44:29.861363 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851355 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 10:44:29.861885 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851360 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 10:44:29.861885 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851364 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 10:44:29.861885 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851368 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 10:44:29.861885 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851372 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 10:44:29.861885 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851379 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 10:44:29.861885 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851383 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 10:44:29.861885 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851389 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 10:44:29.861885 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851393 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 10:44:29.861885 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851397 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 10:44:29.861885 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851403 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 10:44:29.861885 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851408 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 10:44:29.861885 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851412 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 10:44:29.861885 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851416 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 10:44:29.861885 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851420 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 10:44:29.861885 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851424 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 10:44:29.861885 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851443 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 10:44:29.861885 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851448 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 10:44:29.861885 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851452 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 10:44:29.861885 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851457 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 10:44:29.862343 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851462 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 10:44:29.862343 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851465 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 10:44:29.862343 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851469 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 10:44:29.862343 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851474 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 10:44:29.862343 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851478 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 10:44:29.862343 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851482 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 10:44:29.862343 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851487 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 10:44:29.862343 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851491 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 10:44:29.862343 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851496 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 10:44:29.862343 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851502 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 10:44:29.862343 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851508 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 10:44:29.862343 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851512 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 10:44:29.862343 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851517 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 10:44:29.862343 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851521 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 10:44:29.862343 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851525 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 10:44:29.862711 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.851571 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 10:44:29.862711 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851628 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 10:44:29.862711 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851635 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 10:44:29.862711 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851646 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 10:44:29.862711 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851651 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 10:44:29.862711 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851659 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 10:44:29.862711 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851664 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 10:44:29.862711 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851670 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 10:44:29.862711 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851675 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 10:44:29.862711 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851679 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 10:44:29.862711 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851683 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 10:44:29.862711 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851688 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 10:44:29.862711 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851692 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 10:44:29.862711 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851697 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 10:44:29.862711 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851703 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 10:44:29.863081 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851709 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 10:44:29.863081 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851716 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 10:44:29.863081 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851721 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 10:44:29.863081 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851725 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 10:44:29.863081 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851730 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 10:44:29.863081 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851734 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 10:44:29.863081 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851738 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 10:44:29.863081 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851742 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 10:44:29.863081 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851747 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 10:44:29.863081 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851751 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 10:44:29.863081 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851756 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 10:44:29.863081 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851760 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 10:44:29.863081 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851765 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 10:44:29.863081 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851769 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 10:44:29.863081 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851773 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 10:44:29.863081 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851778 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 10:44:29.863081 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851782 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 10:44:29.863081 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851786 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 10:44:29.863081 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851790 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 10:44:29.863081 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851794 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 10:44:29.863560 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851799 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 10:44:29.863560 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851804 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 10:44:29.863560 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851810 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 10:44:29.863560 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851815 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 10:44:29.863560 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851820 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 10:44:29.863560 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851824 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 10:44:29.863560 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851829 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 10:44:29.863560 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851834 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 10:44:29.863560 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851838 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 10:44:29.863560 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851842 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 10:44:29.863560 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851846 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 10:44:29.863560 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851851 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 10:44:29.863560 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851855 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 10:44:29.863560 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851860 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 10:44:29.863560 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851864 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 10:44:29.863560 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851871 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 10:44:29.863560 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851878 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 10:44:29.863560 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851883 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 10:44:29.863560 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851888 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 10:44:29.863560 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851892 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 10:44:29.864036 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851896 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 10:44:29.864036 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851900 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 10:44:29.864036 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851905 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 10:44:29.864036 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851909 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 10:44:29.864036 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851913 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 10:44:29.864036 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851917 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 10:44:29.864036 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851921 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 10:44:29.864036 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851925 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 10:44:29.864036 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851929 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 10:44:29.864036 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851933 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 10:44:29.864036 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851937 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 10:44:29.864036 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851941 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 10:44:29.864036 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851946 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 10:44:29.864036 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851950 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 10:44:29.864036 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851956 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 10:44:29.864036 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851961 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 10:44:29.864036 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851967 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 10:44:29.864036 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851971 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 10:44:29.864036 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851976 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 10:44:29.864036 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851980 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 10:44:29.864517 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851984 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 10:44:29.864517 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851989 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 10:44:29.864517 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851993 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 10:44:29.864517 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.851997 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 10:44:29.864517 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852002 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 10:44:29.864517 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852006 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 10:44:29.864517 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852010 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 10:44:29.864517 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852014 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 10:44:29.864517 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852018 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 10:44:29.864517 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852023 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 10:44:29.864517 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852027 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 10:44:29.864517 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852031 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 10:44:29.864517 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852036 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 10:44:29.864517 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852040 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 10:44:29.864517 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852044 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 10:44:29.864517 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852049 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 10:44:29.864517 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852054 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 10:44:29.864517 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852058 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 10:44:29.864517 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852062 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 10:44:29.864517 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852066 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 10:44:29.864974 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852070 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 10:44:29.864974 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852074 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 10:44:29.864974 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852078 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 10:44:29.864974 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852082 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 10:44:29.864974 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852087 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 10:44:29.864974 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852091 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 10:44:29.864974 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852096 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 10:44:29.864974 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852101 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 10:44:29.864974 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852108 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 10:44:29.864974 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852112 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 10:44:29.864974 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852116 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 10:44:29.864974 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.852120 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 10:44:29.864974 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.858916 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 10:44:29.864974 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.858933 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 10:44:29.864974 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.859629 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 10:44:29.864974 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.859637 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 10:44:29.864974 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859697 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 10:44:29.864974 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859702 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 10:44:29.864974 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859706 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 10:44:29.865438 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859710 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 10:44:29.865438 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859714 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 10:44:29.865438 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859717 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 10:44:29.865438 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859720 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 10:44:29.865438 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859723 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 10:44:29.865438 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859726 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 10:44:29.865438 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859729 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 10:44:29.865438 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859731 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 10:44:29.865438 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859734 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 10:44:29.865438 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859737 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 10:44:29.865438 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859740 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 10:44:29.865438 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859743 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 10:44:29.865438 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859746 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 10:44:29.865438 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859748 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 10:44:29.865438 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859751 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 10:44:29.865438 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859754 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 10:44:29.865438 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859757 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 10:44:29.865438 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859760 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 10:44:29.865438 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859763 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 10:44:29.865438 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859766 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 10:44:29.865920 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859768 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 10:44:29.865920 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859771 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 10:44:29.865920 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859775 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 10:44:29.865920 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859779 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 10:44:29.865920 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859782 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 10:44:29.865920 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859785 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 10:44:29.865920 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859789 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 10:44:29.865920 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859792 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 10:44:29.865920 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859795 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 10:44:29.865920 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859797 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 10:44:29.865920 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859800 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 10:44:29.865920 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859802 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 10:44:29.865920 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859805 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 10:44:29.865920 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859808 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 10:44:29.865920 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859810 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 10:44:29.865920 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859814 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 10:44:29.865920 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859816 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 10:44:29.865920 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859819 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 10:44:29.865920 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859822 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 10:44:29.866353 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859824 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 10:44:29.866353 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859827 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 10:44:29.866353 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859829 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 10:44:29.866353 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859832 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 10:44:29.866353 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859834 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 10:44:29.866353 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859837 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 10:44:29.866353 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859840 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 10:44:29.866353 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859842 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 10:44:29.866353 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859845 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 10:44:29.866353 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859848 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 10:44:29.866353 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859850 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 10:44:29.866353 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859853 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 10:44:29.866353 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859856 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 10:44:29.866353 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859859 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 10:44:29.866353 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859861 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 10:44:29.866353 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859864 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 10:44:29.866353 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859866 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 10:44:29.866353 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859869 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 10:44:29.866353 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859872 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 10:44:29.866353 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859874 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 10:44:29.866848 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859878 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 10:44:29.866848 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859881 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 10:44:29.866848 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859884 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 10:44:29.866848 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859886 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 10:44:29.866848 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859889 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 10:44:29.866848 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859891 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 10:44:29.866848 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859894 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 10:44:29.866848 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859897 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 10:44:29.866848 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859899 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 10:44:29.866848 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859902 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 10:44:29.866848 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859904 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 10:44:29.866848 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859907 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 10:44:29.866848 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859909 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 10:44:29.866848 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859912 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 10:44:29.866848 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859915 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 10:44:29.866848 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859918 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 10:44:29.866848 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859920 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 10:44:29.866848 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859923 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 10:44:29.866848 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859926 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 10:44:29.867765 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859929 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 10:44:29.867765 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859931 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 10:44:29.867765 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859934 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 10:44:29.867765 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859937 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 10:44:29.867765 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859940 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 10:44:29.867765 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859942 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 10:44:29.867765 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859945 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 10:44:29.867765 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859948 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 10:44:29.867765 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859950 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 10:44:29.867765 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859952 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 10:44:29.867765 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859956 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 10:44:29.867765 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859960 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 10:44:29.867765 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859964 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 10:44:29.867765 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859966 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 10:44:29.867765 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859969 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 10:44:29.867765 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859972 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 10:44:29.867765 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859975 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 10:44:29.867765 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859978 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 10:44:29.867765 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859980 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 10:44:29.868479 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859982 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 10:44:29.868479 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859985 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 10:44:29.868479 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859987 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 10:44:29.868479 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859990 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 10:44:29.868479 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859993 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 10:44:29.868479 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.859995 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 10:44:29.868479 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.860024 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 10:44:29.868479 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860202 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 10:44:29.868479 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860206 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 10:44:29.868479 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860210 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 10:44:29.868479 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860213 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 10:44:29.868479 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860216 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 10:44:29.868479 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860219 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 10:44:29.868479 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860222 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 10:44:29.868479 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860224 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 10:44:29.868479 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860226 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 10:44:29.868959 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860229 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 10:44:29.868959 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860231 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 10:44:29.868959 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860234 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 10:44:29.868959 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860237 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 10:44:29.868959 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860239 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 10:44:29.868959 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860241 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 10:44:29.868959 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860246 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 10:44:29.868959 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860250 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 10:44:29.868959 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860254 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 10:44:29.868959 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860257 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 10:44:29.868959 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860260 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 10:44:29.868959 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860263 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 10:44:29.868959 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860266 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 10:44:29.868959 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860268 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 10:44:29.868959 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860271 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 10:44:29.868959 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860273 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 10:44:29.868959 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860276 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 10:44:29.868959 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860278 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 10:44:29.869579 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860281 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 10:44:29.869579 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860284 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 10:44:29.869579 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860286 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 10:44:29.869579 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860289 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 10:44:29.869579 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860291 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 10:44:29.869579 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860294 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 10:44:29.869579 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860296 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 10:44:29.869579 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860299 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 10:44:29.869579 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860301 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 10:44:29.869579 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860304 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 10:44:29.869579 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860306 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 10:44:29.869579 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860309 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 10:44:29.869579 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860311 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 10:44:29.869579 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860314 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 10:44:29.869579 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860316 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 10:44:29.869579 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860319 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 10:44:29.869579 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860321 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 10:44:29.869579 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860325 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 10:44:29.869579 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860327 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 10:44:29.869579 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860329 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 10:44:29.870055 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860332 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 10:44:29.870055 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860335 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 10:44:29.870055 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860337 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 10:44:29.870055 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860340 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 10:44:29.870055 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860342 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 10:44:29.870055 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860345 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 10:44:29.870055 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860347 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 10:44:29.870055 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860350 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 10:44:29.870055 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860353 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 10:44:29.870055 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860355 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 10:44:29.870055 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860357 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 10:44:29.870055 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860360 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 10:44:29.870055 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860362 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 10:44:29.870055 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860365 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 10:44:29.870055 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860367 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 10:44:29.870055 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860370 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 10:44:29.870055 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860372 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 10:44:29.870055 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860375 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 10:44:29.870055 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860377 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 10:44:29.870633 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860380 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 10:44:29.870633 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860382 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 10:44:29.870633 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860384 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 10:44:29.870633 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860387 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 10:44:29.870633 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860389 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 10:44:29.870633 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860392 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 10:44:29.870633 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860394 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 10:44:29.870633 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860397 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 10:44:29.870633 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860400 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 10:44:29.870633 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860402 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 10:44:29.870633 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860405 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 10:44:29.870633 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860407 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 10:44:29.870633 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860410 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 10:44:29.870633 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860412 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 10:44:29.870633 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860415 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 10:44:29.870633 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860417 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 10:44:29.870633 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860420 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 10:44:29.870633 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860423 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 10:44:29.870633 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860437 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 10:44:29.871085 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860440 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 10:44:29.871085 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860443 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 10:44:29.871085 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860446 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 10:44:29.871085 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860449 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 10:44:29.871085 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860451 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 10:44:29.871085 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860454 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 10:44:29.871085 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860456 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 10:44:29.871085 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860459 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 10:44:29.871085 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860461 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 10:44:29.871085 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860464 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 10:44:29.871085 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860466 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 10:44:29.871085 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860469 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 10:44:29.871085 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860471 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 10:44:29.871085 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860474 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 10:44:29.871085 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860476 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 10:44:29.871085 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860479 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 10:44:29.871085 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860481 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 10:44:29.871085 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860484 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 10:44:29.871085 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860487 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 10:44:29.871085 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860489 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 10:44:29.871577 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860492 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 10:44:29.871577 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.860515 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 10:44:29.871577 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860547 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 10:44:29.871577 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860551 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 10:44:29.871577 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860554 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 10:44:29.871577 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860557 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 10:44:29.871577 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860559 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 10:44:29.871577 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860562 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 10:44:29.871577 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860564 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 10:44:29.871577 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860567 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 10:44:29.871577 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860569 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 10:44:29.871577 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860572 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 10:44:29.871577 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860575 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 10:44:29.871577 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860577 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 10:44:29.871577 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860580 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 10:44:29.871935 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860583 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 10:44:29.871935 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860586 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 10:44:29.871935 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860588 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 10:44:29.871935 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860591 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 10:44:29.871935 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860593 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 10:44:29.871935 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860596 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 10:44:29.871935 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860598 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 10:44:29.871935 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860601 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 10:44:29.871935 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860603 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 10:44:29.871935 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860606 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 10:44:29.871935 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860608 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 10:44:29.871935 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860611 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 10:44:29.871935 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860613 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 10:44:29.871935 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860616 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 10:44:29.871935 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860618 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 10:44:29.871935 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860621 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 10:44:29.871935 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860623 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 10:44:29.871935 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860626 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 10:44:29.871935 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860628 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 10:44:29.871935 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860631 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 10:44:29.872395 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860634 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 10:44:29.872395 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860636 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 10:44:29.872395 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860639 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 10:44:29.872395 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860641 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 10:44:29.872395 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860644 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 10:44:29.872395 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860647 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 10:44:29.872395 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860649 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 10:44:29.872395 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860652 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 10:44:29.872395 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860654 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 10:44:29.872395 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860656 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 10:44:29.872395 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860659 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 10:44:29.872395 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860661 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 10:44:29.872395 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860664 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 10:44:29.872395 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860668 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 10:44:29.872395 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860672 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 10:44:29.872395 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860676 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 10:44:29.872395 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860680 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 10:44:29.872395 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860682 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 10:44:29.872395 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860686 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 10:44:29.872846 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860688 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 10:44:29.872846 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860691 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 10:44:29.872846 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860694 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 10:44:29.872846 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860696 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 10:44:29.872846 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860699 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 10:44:29.872846 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860701 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 10:44:29.872846 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860704 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 10:44:29.872846 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860707 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 10:44:29.872846 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860710 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 10:44:29.872846 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860712 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 10:44:29.872846 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860715 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 10:44:29.872846 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860717 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 10:44:29.872846 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860720 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 10:44:29.872846 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860722 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 10:44:29.872846 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860725 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 10:44:29.872846 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860728 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 10:44:29.872846 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860730 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 10:44:29.872846 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860732 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 10:44:29.872846 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860735 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 10:44:29.872846 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860737 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 10:44:29.873304 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860740 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 10:44:29.873304 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860742 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 10:44:29.873304 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860745 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 10:44:29.873304 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860748 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 10:44:29.873304 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860750 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 10:44:29.873304 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860753 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 10:44:29.873304 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860755 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 10:44:29.873304 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860763 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 10:44:29.873304 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860765 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 10:44:29.873304 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860768 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 10:44:29.873304 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860771 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 10:44:29.873304 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860773 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 10:44:29.873304 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860776 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 10:44:29.873304 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860778 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 10:44:29.873304 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860781 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 10:44:29.873304 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860784 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 10:44:29.873304 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860786 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 10:44:29.873304 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860789 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 10:44:29.873304 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860791 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 10:44:29.873304 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860794 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 10:44:29.873804 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860797 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 10:44:29.873804 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860799 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 10:44:29.873804 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860802 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 10:44:29.873804 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860805 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 10:44:29.873804 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860807 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 10:44:29.873804 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860810 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 10:44:29.873804 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860812 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 10:44:29.873804 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860815 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 10:44:29.873804 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860817 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 10:44:29.873804 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860820 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 10:44:29.873804 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860824 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 10:44:29.873804 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860826 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 10:44:29.873804 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860829 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 10:44:29.873804 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:29.860831 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 10:44:29.873804 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.860890 2579 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 10:44:29.873804 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.862802 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 10:44:29.873804 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.863538 2579 server.go:1014] "Starting client certificate rotation" Oct 30 10:44:29.873804 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.863634 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 10:44:29.873804 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.863674 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 10:44:29.874269 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.865509 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 10:44:29.874269 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.869318 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 10:44:29.879284 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.879267 2579 log.go:25] "Validated CRI v1 runtime API" Oct 30 10:44:29.882226 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.882212 2579 log.go:25] "Validated CRI v1 image API" Oct 30 10:44:29.883072 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.883048 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 10:44:29.886132 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.886111 2579 fs.go:135] Filesystem UUIDs: map[65f273d2-6938-4a11-89d1-a50ef37222b7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9d148982-660c-4fa9-88e7-bc2b9bae4cb1:/dev/nvme0n1p4] Oct 30 10:44:29.886190 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.886129 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 10:44:29.891602 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.891489 2579 manager.go:217] Machine: {Timestamp:2026-10-30 10:44:29.889700593 +0000 UTC m=+0.311715884 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2499996 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28d00138eb7f563a421dcdff06e4ed SystemUUID:ec28d001-38eb-7f56-3a42-1dcdff06e4ed BootID:3139dbc3-b461-4817-b629-2ae03f4b7682 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cb:0c:21:a3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cb:0c:21:a3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:22:b4:a1:05:cd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 10:44:29.891735 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.891604 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 10:44:29.891735 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.891673 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 10:44:29.894382 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.894352 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 10:44:29.894566 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.894384 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-242.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 10:44:29.894566 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.894568 2579 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 10:44:29.894679 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.894577 2579 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 10:44:29.894679 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.894604 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 10:44:29.894679 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.894616 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 10:44:29.895285 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.895274 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 10:44:29.898732 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.898720 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 10:44:29.898847 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.898838 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 10:44:29.899464 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.899448 2579 kubelet.go:493] "Attempting to sync node with API server" Oct 30 10:44:29.899464 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.899461 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 10:44:29.899606 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.899471 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 10:44:29.899606 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.899480 2579 kubelet.go:405] "Adding apiserver pod source" Oct 30 10:44:29.899606 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.899489 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 10:44:29.899838 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.899826 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 10:44:29.902086 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.902061 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 10:44:29.905674 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.905654 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 10:44:29.906872 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.906852 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 10:44:29.906932 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.906904 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 10:44:29.907588 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.907565 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 10:44:29.907588 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.907588 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 10:44:29.907690 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.907598 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 10:44:29.907690 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.907608 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 10:44:29.907690 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.907618 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 10:44:29.907690 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.907627 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 10:44:29.908063 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.908053 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 10:44:29.908094 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.908067 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 10:44:29.908094 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.908078 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 10:44:29.908094 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.908087 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 10:44:29.908188 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.908101 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 10:44:29.908188 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.908117 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 10:44:29.908686 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.908673 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 10:44:29.908724 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.908690 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 10:44:29.912329 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.912314 2579 server.go:1263] "Started kubelet" Oct 30 10:44:29.912410 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.912381 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 10:44:29.912493 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.912395 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 10:44:29.912493 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.912480 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 10:44:29.912977 ip-10-0-137-242 systemd[1]: Started Kubernetes Kubelet. Oct 30 10:44:29.913808 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.913793 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 10:44:29.914566 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.914552 2579 server.go:317] "Adding debug handlers to kubelet server" Oct 30 10:44:29.918829 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:29.918811 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 10:44:29.919979 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.919961 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 10:44:29.920378 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.920360 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 10:44:29.920499 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:29.920480 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Oct 30 10:44:29.921017 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.920959 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 10:44:29.921017 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.920989 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 10:44:29.921017 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.921003 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 10:44:29.921177 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.921164 2579 reconstruct.go:97] "Volume reconstruction finished" Oct 30 10:44:29.921177 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.921172 2579 reconciler.go:29] "Reconciler: start to sync state" Oct 30 10:44:29.921270 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.921245 2579 factory.go:55] Registering systemd factory Oct 30 10:44:29.921316 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.921271 2579 factory.go:223] Registration of the systemd container factory successfully Oct 30 10:44:29.921541 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.921520 2579 factory.go:153] Registering CRI-O factory Oct 30 10:44:29.921541 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.921535 2579 factory.go:223] Registration of the crio container factory successfully Oct 30 10:44:29.921685 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.921592 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 10:44:29.921685 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.921610 2579 factory.go:103] Registering Raw factory Oct 30 10:44:29.921685 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.921626 2579 manager.go:1199] Started watching for new ooms in manager Oct 30 10:44:29.922415 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.922397 2579 manager.go:319] Starting recovery of all containers Oct 30 10:44:29.932627 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.932607 2579 manager.go:324] Recovery completed Oct 30 10:44:29.937213 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.937200 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 10:44:29.942533 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.942501 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientMemory" Oct 30 10:44:29.942621 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.942540 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 10:44:29.942621 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.942556 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientPID" Oct 30 10:44:29.943132 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.943116 2579 cpu_manager.go:226] "Starting" policy="none" Oct 30 10:44:29.943132 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.943125 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 10:44:29.943132 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.943129 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 10:44:29.943294 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.943150 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 10:44:29.943801 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.943783 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientMemory" Oct 30 10:44:29.943885 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.943805 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 10:44:29.943885 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.943814 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientPID" Oct 30 10:44:29.945564 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.945549 2579 policy_none.go:50] "Start" Oct 30 10:44:29.945636 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.945567 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 10:44:29.945636 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.945579 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 10:44:29.947422 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.947411 2579 policy_none.go:44] "Start" Oct 30 10:44:29.948808 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:29.948779 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-242.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 10:44:29.948888 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.948779 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-242.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 10:44:29.949546 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:29.948763 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-242.ec2.internal.18e348c0e38e9f45 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-242.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-242.ec2.internal,},FirstTimestamp:2026-10-30 10:44:29.912284997 +0000 UTC m=+0.334300287,LastTimestamp:2026-10-30 10:44:29.912284997 +0000 UTC m=+0.334300287,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-242.ec2.internal,}" Oct 30 10:44:29.963184 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.963168 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 10:44:29.963270 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.963204 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6wcrc" Oct 30 10:44:29.965917 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:29.965895 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 10:44:29.965917 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:29.965906 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 10:44:29.966015 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:29.965972 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-242.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 10:44:29.966202 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:29.966125 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-242.ec2.internal.18e348c0e55c16ba default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-242.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-242.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-242.ec2.internal,},FirstTimestamp:2026-10-30 10:44:29.942527674 +0000 UTC m=+0.364542964,LastTimestamp:2026-10-30 10:44:29.942527674 +0000 UTC m=+0.364542964,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-242.ec2.internal,}" Oct 30 10:44:29.981039 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.981012 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 10:44:29.983549 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.983534 2579 manager.go:341] "Starting Device Plugin manager" Oct 30 10:44:29.984087 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:29.984072 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 10:44:29.984146 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.984091 2579 server.go:87] "Starting device plugin registration server" Oct 30 10:44:29.984328 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.984315 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 10:44:29.984392 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.984329 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 10:44:29.984454 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.984403 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 10:44:29.984502 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.984496 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 10:44:29.984545 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:29.984504 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 10:44:29.985025 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:29.985004 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 10:44:29.985097 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:29.985056 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-242.ec2.internal\" not found" Oct 30 10:44:30.042342 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.042310 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 10:44:30.043532 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.043514 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 10:44:30.043631 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.043538 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 10:44:30.043631 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.043554 2579 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 10:44:30.043631 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:30.043584 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 10:44:30.056808 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.056793 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 10:44:30.084839 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.084825 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 10:44:30.089003 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.088987 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientMemory" Oct 30 10:44:30.089095 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.089019 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 10:44:30.089095 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.089033 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientPID" Oct 30 10:44:30.089095 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.089064 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-242.ec2.internal" Oct 30 10:44:30.104867 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.104823 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-242.ec2.internal" Oct 30 10:44:30.104867 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:30.104844 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-242.ec2.internal\": node \"ip-10-0-137-242.ec2.internal\" not found" Oct 30 10:44:30.144393 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.144373 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal"] Oct 30 10:44:30.144526 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.144453 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 10:44:30.145324 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.145291 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientMemory" Oct 30 10:44:30.145324 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.145321 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 10:44:30.145324 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.145336 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientPID" Oct 30 10:44:30.145610 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:30.145588 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Oct 30 10:44:30.147218 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.147203 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 10:44:30.147347 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.147335 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Oct 30 10:44:30.147393 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.147363 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 10:44:30.147935 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.147920 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientMemory" Oct 30 10:44:30.148020 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.147950 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 10:44:30.148020 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.147926 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientMemory" Oct 30 10:44:30.148020 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.147961 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientPID" Oct 30 10:44:30.148020 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.147984 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 10:44:30.148020 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.148001 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientPID" Oct 30 10:44:30.149732 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.149716 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Oct 30 10:44:30.149814 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.149747 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 10:44:30.150474 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.150460 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientMemory" Oct 30 10:44:30.150542 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.150486 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 10:44:30.150542 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.150495 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientPID" Oct 30 10:44:30.185394 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:30.185373 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-242.ec2.internal\" not found" node="ip-10-0-137-242.ec2.internal" Oct 30 10:44:30.188474 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:30.188456 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-242.ec2.internal\" not found" node="ip-10-0-137-242.ec2.internal" Oct 30 10:44:30.222440 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.222407 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d39640f4fcc35209a29f428a929973b1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" (UID: \"d39640f4fcc35209a29f428a929973b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Oct 30 10:44:30.222530 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.222442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d39640f4fcc35209a29f428a929973b1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" (UID: \"d39640f4fcc35209a29f428a929973b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Oct 30 10:44:30.222530 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.222458 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6dc9a86d1bbc0350320467df988d567a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-242.ec2.internal\" (UID: \"6dc9a86d1bbc0350320467df988d567a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Oct 30 10:44:30.246343 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:30.246323 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Oct 30 10:44:30.323515 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.323492 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6dc9a86d1bbc0350320467df988d567a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-242.ec2.internal\" (UID: \"6dc9a86d1bbc0350320467df988d567a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Oct 30 10:44:30.323612 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.323516 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d39640f4fcc35209a29f428a929973b1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" (UID: \"d39640f4fcc35209a29f428a929973b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Oct 30 10:44:30.323612 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.323540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d39640f4fcc35209a29f428a929973b1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" (UID: \"d39640f4fcc35209a29f428a929973b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Oct 30 10:44:30.323612 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.323569 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d39640f4fcc35209a29f428a929973b1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" (UID: \"d39640f4fcc35209a29f428a929973b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Oct 30 10:44:30.323612 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.323575 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6dc9a86d1bbc0350320467df988d567a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-242.ec2.internal\" (UID: \"6dc9a86d1bbc0350320467df988d567a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Oct 30 10:44:30.323612 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.323609 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d39640f4fcc35209a29f428a929973b1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" (UID: \"d39640f4fcc35209a29f428a929973b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Oct 30 10:44:30.347285 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:30.347261 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Oct 30 10:44:30.447542 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:30.447464 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Oct 30 10:44:30.488680 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.488641 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Oct 30 10:44:30.491779 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.491758 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Oct 30 10:44:30.548631 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:30.548486 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Oct 30 10:44:30.649008 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:30.648977 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Oct 30 10:44:30.749421 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:30.749361 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Oct 30 10:44:30.849889 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:30.849859 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Oct 30 10:44:30.864027 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.864004 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 10:44:30.864213 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.864170 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 10:44:30.950780 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:30.950743 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Oct 30 10:44:30.959738 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.959713 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 10:44:30.967092 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.967046 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 10:39:30 +0000 UTC" deadline="2028-07-03 04:49:09.30146525 +0000 UTC" Oct 30 10:44:30.967092 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:30.967092 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14682h4m38.334378042s" Oct 30 10:44:31.051579 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:31.051553 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Oct 30 10:44:31.053637 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:31.053604 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc9a86d1bbc0350320467df988d567a.slice/crio-0d54cc91ee15ed4118ea45d0d6a1cb02b727905f2203c2637af7dc57120f9a67 WatchSource:0}: Error finding container 0d54cc91ee15ed4118ea45d0d6a1cb02b727905f2203c2637af7dc57120f9a67: Status 404 returned error can't find the container with id 0d54cc91ee15ed4118ea45d0d6a1cb02b727905f2203c2637af7dc57120f9a67 Oct 30 10:44:31.060672 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.060653 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 10:44:31.064723 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:31.064699 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd39640f4fcc35209a29f428a929973b1.slice/crio-2f0a9655b588d3a8d23f1e9090caf5b0a8c49ac951b41013762ce64399de158a WatchSource:0}: Error finding container 2f0a9655b588d3a8d23f1e9090caf5b0a8c49ac951b41013762ce64399de158a: Status 404 returned error can't find the container with id 2f0a9655b588d3a8d23f1e9090caf5b0a8c49ac951b41013762ce64399de158a Oct 30 10:44:31.086833 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.086813 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 10:44:31.087733 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.087716 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientMemory" Oct 30 10:44:31.087805 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.087746 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 10:44:31.087805 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.087757 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientPID" Oct 30 10:44:31.153139 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:31.153116 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Oct 30 10:44:31.254265 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:31.254236 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Oct 30 10:44:31.355027 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:31.354962 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Oct 30 10:44:31.358928 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.358911 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 10:44:31.421063 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.421035 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Oct 30 10:44:31.438892 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.438870 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 10:44:31.445012 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.444992 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 10:44:31.446316 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.446302 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Oct 30 10:44:31.470545 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.470528 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 10:44:31.900710 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.900639 2579 apiserver.go:52] "Watching apiserver" Oct 30 10:44:31.920133 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.920104 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 10:44:31.933483 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.933438 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 10:44:31.933883 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.933849 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-8xmhg","openshift-multus/multus-ftv4s","openshift-multus/network-metrics-daemon-4cgfp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq","openshift-cluster-node-tuning-operator/tuned-rpqq9","openshift-network-diagnostics/network-check-target-fk5zl","openshift-network-operator/iptables-alerter-vd8gp","openshift-image-registry/node-ca-tvrzn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-skmk7","kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal","openshift-dns/node-resolver-gll67","openshift-multus/multus-additional-cni-plugins-98jtw","kube-system/global-pull-secret-syncer-vskj5"] Oct 30 10:44:31.937467 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.936671 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vd8gp" Oct 30 10:44:31.939129 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.939064 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:44:31.941724 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.941454 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:31.941724 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:31.941529 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cgfp" podUID="b06a8e3b-6db3-4cc7-b7e2-dc14122245ce" Oct 30 10:44:31.943970 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.943946 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:44:31.946975 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.946450 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:31.949143 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.949120 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:44:31.949238 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:31.949205 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fk5zl" podUID="e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8" Oct 30 10:44:31.951283 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.951261 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:31.951380 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:31.951331 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vskj5" podUID="23abd71b-1521-4aec-9d79-198f0f8e14a2" Oct 30 10:44:31.954318 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.953989 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8xmhg" Oct 30 10:44:31.954318 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.954225 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 10:44:31.954495 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.954472 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 10:44:31.954569 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.954547 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 10:44:31.954685 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.954658 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9pzxh\"" Oct 30 10:44:31.956733 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.956712 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ftv4s" Oct 30 10:44:31.958072 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.958053 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 10:44:31.958201 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.958186 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 10:44:31.958301 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.958287 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 10:44:31.959039 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.959011 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lntjm\"" Oct 30 10:44:31.959108 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.959072 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 10:44:31.959190 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.959177 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 10:44:31.962693 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.962669 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:31.964044 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.963981 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-472bd\"" Oct 30 10:44:31.964044 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.964004 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 10:44:31.964044 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.964004 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 10:44:31.964252 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.964074 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 10:44:31.964396 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.964378 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 10:44:31.964593 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.964577 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 10:44:31.964729 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.964703 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s5t5b\"" Oct 30 10:44:31.965436 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.965414 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tvrzn" Oct 30 10:44:31.967932 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.967914 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gll67" Oct 30 10:44:31.986653 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.986390 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 10:44:31.986653 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:31.986484 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5cjrr" Oct 30 10:44:32.022558 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.022540 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 10:44:32.033970 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.033940 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-etc-selinux\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:44:32.034084 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.033978 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-etc-openvswitch\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.034084 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034002 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63196a40-0beb-4a8b-b396-0a78f57560f3-ovn-node-metrics-cert\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.034084 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034066 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs\") pod \"network-metrics-daemon-4cgfp\" (UID: \"b06a8e3b-6db3-4cc7-b7e2-dc14122245ce\") " pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:32.034227 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034118 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-lib-modules\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.034227 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034211 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4b288c6-5317-410f-9f88-0352f4164d20-tmp\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.034298 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034247 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-run-k8s-cni-cncf-io\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.034298 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034275 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-hostroot\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.034362 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034318 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pzwb\" (UniqueName: \"kubernetes.io/projected/9f26208e-e65a-483d-ba73-d45fe31084bc-kube-api-access-4pzwb\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:44:32.034445 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034357 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-sysconfig\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.034445 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034392 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-var-lib-kubelet\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.034445 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034419 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-run-multus-certs\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.034574 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034462 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-run-ovn-kubernetes\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.034574 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034486 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63196a40-0beb-4a8b-b396-0a78f57560f3-ovnkube-script-lib\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.034574 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034506 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-kubernetes\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.034574 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034532 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv7hl\" (UniqueName: \"kubernetes.io/projected/7437f570-02f5-4131-8b0b-140ed029a668-kube-api-access-pv7hl\") pod \"node-ca-tvrzn\" (UID: \"7437f570-02f5-4131-8b0b-140ed029a668\") " pod="openshift-image-registry/node-ca-tvrzn" Oct 30 10:44:32.034731 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034575 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqjpq\" (UniqueName: \"kubernetes.io/projected/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-kube-api-access-wqjpq\") pod \"network-metrics-daemon-4cgfp\" (UID: \"b06a8e3b-6db3-4cc7-b7e2-dc14122245ce\") " pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:32.034731 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034593 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/275105df-bf86-436c-a464-890b4510c09d-system-cni-dir\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:44:32.034731 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034617 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-kubelet\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.034731 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034631 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-systemd-units\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.034731 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034654 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63196a40-0beb-4a8b-b396-0a78f57560f3-ovnkube-config\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.034731 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c64ab741-bc4f-4006-932a-214cf53b6dc3-hosts-file\") pod \"node-resolver-gll67\" (UID: \"c64ab741-bc4f-4006-932a-214cf53b6dc3\") " pod="openshift-dns/node-resolver-gll67" Oct 30 10:44:32.034731 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034718 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/45d9d286-4a6a-4ebb-91ab-324335c4a2ab-iptables-alerter-script\") pod \"iptables-alerter-vd8gp\" (UID: \"45d9d286-4a6a-4ebb-91ab-324335c4a2ab\") " pod="openshift-network-operator/iptables-alerter-vd8gp" Oct 30 10:44:32.034953 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034741 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-host\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.034953 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034823 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-run\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.034953 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034853 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-sys-fs\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:44:32.034953 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034877 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9-agent-certs\") pod \"konnectivity-agent-8xmhg\" (UID: \"fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9\") " pod="kube-system/konnectivity-agent-8xmhg" Oct 30 10:44:32.034953 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034901 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-cnibin\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.034953 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034924 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-run-netns\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.034953 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7437f570-02f5-4131-8b0b-140ed029a668-host\") pod \"node-ca-tvrzn\" (UID: \"7437f570-02f5-4131-8b0b-140ed029a668\") " pod="openshift-image-registry/node-ca-tvrzn" Oct 30 10:44:32.035203 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034970 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7437f570-02f5-4131-8b0b-140ed029a668-serviceca\") pod \"node-ca-tvrzn\" (UID: \"7437f570-02f5-4131-8b0b-140ed029a668\") " pod="openshift-image-registry/node-ca-tvrzn" Oct 30 10:44:32.035203 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.034996 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/275105df-bf86-436c-a464-890b4510c09d-cni-binary-copy\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:44:32.035203 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035021 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-var-lib-openvswitch\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.035203 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035043 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-log-socket\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.035203 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035068 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.035203 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035095 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8pl4\" (UniqueName: \"kubernetes.io/projected/45d9d286-4a6a-4ebb-91ab-324335c4a2ab-kube-api-access-q8pl4\") pod \"iptables-alerter-vd8gp\" (UID: \"45d9d286-4a6a-4ebb-91ab-324335c4a2ab\") " pod="openshift-network-operator/iptables-alerter-vd8gp" Oct 30 10:44:32.035203 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035129 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-var-lib-cni-bin\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.035203 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035154 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-registration-dir\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:44:32.035203 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035178 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-run-netns\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.035203 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035203 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-run-openvswitch\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.035643 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035315 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-device-dir\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:44:32.035643 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035343 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-sys\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.035643 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035373 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp4mk\" (UniqueName: \"kubernetes.io/projected/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8-kube-api-access-lp4mk\") pod \"network-check-target-fk5zl\" (UID: \"e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8\") " pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:44:32.035643 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035400 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9-konnectivity-ca\") pod \"konnectivity-agent-8xmhg\" (UID: \"fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9\") " pod="kube-system/konnectivity-agent-8xmhg" Oct 30 10:44:32.035643 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035426 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-multus-socket-dir-parent\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.035643 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035451 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-var-lib-cni-multus\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.035643 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035476 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrb2t\" (UniqueName: \"kubernetes.io/projected/538de0b2-f736-4723-92d6-e600b9de4a26-kube-api-access-xrb2t\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.035643 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035502 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/275105df-bf86-436c-a464-890b4510c09d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:44:32.035643 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035546 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-run-systemd\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.035643 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035571 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58d4h\" (UniqueName: \"kubernetes.io/projected/63196a40-0beb-4a8b-b396-0a78f57560f3-kube-api-access-58d4h\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.035643 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035636 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-sysctl-d\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.036067 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035680 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/275105df-bf86-436c-a464-890b4510c09d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:44:32.036067 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035738 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzt9k\" (UniqueName: \"kubernetes.io/projected/275105df-bf86-436c-a464-890b4510c09d-kube-api-access-rzt9k\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:44:32.036067 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-node-log\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.036067 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035809 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-cni-bin\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.036067 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035836 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4b288c6-5317-410f-9f88-0352f4164d20-etc-tuned\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.036067 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035873 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-var-lib-kubelet\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.036067 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035964 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:44:32.036067 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.035998 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bflhl\" (UniqueName: \"kubernetes.io/projected/c64ab741-bc4f-4006-932a-214cf53b6dc3-kube-api-access-bflhl\") pod \"node-resolver-gll67\" (UID: \"c64ab741-bc4f-4006-932a-214cf53b6dc3\") " pod="openshift-dns/node-resolver-gll67" Oct 30 10:44:32.036067 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.036025 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-modprobe-d\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.036067 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.036065 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9g8f\" (UniqueName: \"kubernetes.io/projected/c4b288c6-5317-410f-9f88-0352f4164d20-kube-api-access-k9g8f\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.036443 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.036094 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-multus-conf-dir\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.036443 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.036119 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/275105df-bf86-436c-a464-890b4510c09d-cnibin\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:44:32.036443 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.036146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/275105df-bf86-436c-a464-890b4510c09d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:44:32.036443 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.036196 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-slash\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.036443 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.036225 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-run-ovn\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.036443 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.036249 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-cni-netd\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.036443 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.036274 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64ab741-bc4f-4006-932a-214cf53b6dc3-tmp-dir\") pod \"node-resolver-gll67\" (UID: \"c64ab741-bc4f-4006-932a-214cf53b6dc3\") " pod="openshift-dns/node-resolver-gll67" Oct 30 10:44:32.036443 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.036298 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/23abd71b-1521-4aec-9d79-198f0f8e14a2-kubelet-config\") pod \"global-pull-secret-syncer-vskj5\" (UID: \"23abd71b-1521-4aec-9d79-198f0f8e14a2\") " pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:32.036443 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.036329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/23abd71b-1521-4aec-9d79-198f0f8e14a2-dbus\") pod \"global-pull-secret-syncer-vskj5\" (UID: \"23abd71b-1521-4aec-9d79-198f0f8e14a2\") " pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:32.036443 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.036353 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63196a40-0beb-4a8b-b396-0a78f57560f3-env-overrides\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.036443 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.036377 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-multus-cni-dir\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.036443 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.036446 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-etc-kubernetes\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.036926 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.036488 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-sysctl-conf\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.036926 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.036514 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/23abd71b-1521-4aec-9d79-198f0f8e14a2-original-pull-secret\") pod \"global-pull-secret-syncer-vskj5\" (UID: \"23abd71b-1521-4aec-9d79-198f0f8e14a2\") " pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:32.036926 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.036538 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45d9d286-4a6a-4ebb-91ab-324335c4a2ab-host-slash\") pod \"iptables-alerter-vd8gp\" (UID: \"45d9d286-4a6a-4ebb-91ab-324335c4a2ab\") " pod="openshift-network-operator/iptables-alerter-vd8gp" Oct 30 10:44:32.036926 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.036565 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-systemd\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.036926 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.036589 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/538de0b2-f736-4723-92d6-e600b9de4a26-cni-binary-copy\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.036926 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.036615 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/538de0b2-f736-4723-92d6-e600b9de4a26-multus-daemon-config\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.036926 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.036640 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-socket-dir\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:44:32.036926 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.036666 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-system-cni-dir\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.049325 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.049275 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" event={"ID":"d39640f4fcc35209a29f428a929973b1","Type":"ContainerStarted","Data":"2f0a9655b588d3a8d23f1e9090caf5b0a8c49ac951b41013762ce64399de158a"} Oct 30 10:44:32.050734 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.050700 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" event={"ID":"6dc9a86d1bbc0350320467df988d567a","Type":"ContainerStarted","Data":"0d54cc91ee15ed4118ea45d0d6a1cb02b727905f2203c2637af7dc57120f9a67"} Oct 30 10:44:32.063575 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.063486 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 10:44:32.063697 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.063579 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 10:44:32.063697 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.063602 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 10:44:32.063697 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.063683 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ghh8s\"" Oct 30 10:44:32.063941 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.063707 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 10:44:32.063941 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.063695 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 10:44:32.063941 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.063815 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 10:44:32.064131 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.063941 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 10:44:32.064131 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.063956 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-mhmcf\"" Oct 30 10:44:32.064131 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.063975 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 10:44:32.064131 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.063944 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5fdc6\"" Oct 30 10:44:32.064131 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.064009 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 10:44:32.064131 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.064016 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 10:44:32.064131 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.064125 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 10:44:32.064540 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.064138 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 10:44:32.064540 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.064316 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xt7nj\"" Oct 30 10:44:32.064540 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.064374 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 10:44:32.064681 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.064618 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vwt65\"" Oct 30 10:44:32.064681 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.064652 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 10:44:32.137543 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137496 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:44:32.137543 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137538 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bflhl\" (UniqueName: \"kubernetes.io/projected/c64ab741-bc4f-4006-932a-214cf53b6dc3-kube-api-access-bflhl\") pod \"node-resolver-gll67\" (UID: \"c64ab741-bc4f-4006-932a-214cf53b6dc3\") " pod="openshift-dns/node-resolver-gll67" Oct 30 10:44:32.137787 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137557 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-modprobe-d\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.137787 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137581 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9g8f\" (UniqueName: \"kubernetes.io/projected/c4b288c6-5317-410f-9f88-0352f4164d20-kube-api-access-k9g8f\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.137787 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137604 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-multus-conf-dir\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.137787 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137623 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:44:32.137787 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137632 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/275105df-bf86-436c-a464-890b4510c09d-cnibin\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:44:32.137787 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137654 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/275105df-bf86-436c-a464-890b4510c09d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:44:32.137787 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137674 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-slash\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.137787 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137697 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-run-ovn\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.137787 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137720 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-cni-netd\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.137787 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137743 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64ab741-bc4f-4006-932a-214cf53b6dc3-tmp-dir\") pod \"node-resolver-gll67\" (UID: \"c64ab741-bc4f-4006-932a-214cf53b6dc3\") " pod="openshift-dns/node-resolver-gll67" Oct 30 10:44:32.137787 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137747 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-modprobe-d\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.137787 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137767 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/23abd71b-1521-4aec-9d79-198f0f8e14a2-kubelet-config\") pod \"global-pull-secret-syncer-vskj5\" (UID: \"23abd71b-1521-4aec-9d79-198f0f8e14a2\") " pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:32.137787 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137786 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/23abd71b-1521-4aec-9d79-198f0f8e14a2-dbus\") pod \"global-pull-secret-syncer-vskj5\" (UID: \"23abd71b-1521-4aec-9d79-198f0f8e14a2\") " pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:32.138410 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137804 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63196a40-0beb-4a8b-b396-0a78f57560f3-env-overrides\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.138410 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137820 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-multus-cni-dir\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.138410 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137843 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-etc-kubernetes\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.138410 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137867 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-sysctl-conf\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.138410 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/23abd71b-1521-4aec-9d79-198f0f8e14a2-original-pull-secret\") pod \"global-pull-secret-syncer-vskj5\" (UID: \"23abd71b-1521-4aec-9d79-198f0f8e14a2\") " pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:32.138410 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137901 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45d9d286-4a6a-4ebb-91ab-324335c4a2ab-host-slash\") pod \"iptables-alerter-vd8gp\" (UID: \"45d9d286-4a6a-4ebb-91ab-324335c4a2ab\") " pod="openshift-network-operator/iptables-alerter-vd8gp" Oct 30 10:44:32.138410 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137932 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-systemd\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.138410 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137948 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/538de0b2-f736-4723-92d6-e600b9de4a26-cni-binary-copy\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.138410 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137950 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/23abd71b-1521-4aec-9d79-198f0f8e14a2-kubelet-config\") pod \"global-pull-secret-syncer-vskj5\" (UID: \"23abd71b-1521-4aec-9d79-198f0f8e14a2\") " pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:32.138410 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137972 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/538de0b2-f736-4723-92d6-e600b9de4a26-multus-daemon-config\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.138410 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137996 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-socket-dir\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:44:32.138410 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.137996 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-run-ovn\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.138410 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.138019 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-slash\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.138410 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.138022 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-multus-conf-dir\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.138410 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.138084 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-systemd\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.138410 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:32.138124 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 10:44:32.138410 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.138131 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-system-cni-dir\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.138410 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.138175 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45d9d286-4a6a-4ebb-91ab-324335c4a2ab-host-slash\") pod \"iptables-alerter-vd8gp\" (UID: \"45d9d286-4a6a-4ebb-91ab-324335c4a2ab\") " pod="openshift-network-operator/iptables-alerter-vd8gp" Oct 30 10:44:32.139285 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:32.138204 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23abd71b-1521-4aec-9d79-198f0f8e14a2-original-pull-secret podName:23abd71b-1521-4aec-9d79-198f0f8e14a2 nodeName:}" failed. No retries permitted until 2026-10-30 10:44:32.638180144 +0000 UTC m=+3.060195447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/23abd71b-1521-4aec-9d79-198f0f8e14a2-original-pull-secret") pod "global-pull-secret-syncer-vskj5" (UID: "23abd71b-1521-4aec-9d79-198f0f8e14a2") : object "kube-system"/"original-pull-secret" not registered Oct 30 10:44:32.139285 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.138215 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-cni-netd\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.139285 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.138020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-system-cni-dir\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.139285 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.138248 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-etc-selinux\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:44:32.139285 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.138265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-etc-openvswitch\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.139285 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.138281 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63196a40-0beb-4a8b-b396-0a78f57560f3-ovn-node-metrics-cert\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.139285 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.138297 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs\") pod \"network-metrics-daemon-4cgfp\" (UID: \"b06a8e3b-6db3-4cc7-b7e2-dc14122245ce\") " pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:32.139285 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.138312 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-lib-modules\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.139285 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.138325 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/23abd71b-1521-4aec-9d79-198f0f8e14a2-dbus\") pod \"global-pull-secret-syncer-vskj5\" (UID: \"23abd71b-1521-4aec-9d79-198f0f8e14a2\") " pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:32.139285 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.138327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4b288c6-5317-410f-9f88-0352f4164d20-tmp\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.139285 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.138386 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-etc-selinux\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:44:32.139285 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.138409 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-etc-openvswitch\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.139285 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.138562 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/275105df-bf86-436c-a464-890b4510c09d-cnibin\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:44:32.139285 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.138718 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-sysctl-conf\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.139285 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.138740 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-multus-cni-dir\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.139285 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.139228 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/275105df-bf86-436c-a464-890b4510c09d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:44:32.139285 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.139275 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-etc-kubernetes\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.139850 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.139329 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-run-k8s-cni-cncf-io\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.139850 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.139365 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-hostroot\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.139850 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.139398 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pzwb\" (UniqueName: \"kubernetes.io/projected/9f26208e-e65a-483d-ba73-d45fe31084bc-kube-api-access-4pzwb\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:44:32.139850 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.139433 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-sysconfig\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.139850 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.139463 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-var-lib-kubelet\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.139850 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.139507 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-run-multus-certs\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.139850 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.139508 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64ab741-bc4f-4006-932a-214cf53b6dc3-tmp-dir\") pod \"node-resolver-gll67\" (UID: \"c64ab741-bc4f-4006-932a-214cf53b6dc3\") " pod="openshift-dns/node-resolver-gll67" Oct 30 10:44:32.139850 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.139563 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-run-multus-certs\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.139850 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.139589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-run-ovn-kubernetes\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.139850 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.139607 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-run-k8s-cni-cncf-io\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.139850 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.139630 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63196a40-0beb-4a8b-b396-0a78f57560f3-ovnkube-script-lib\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.139850 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.139648 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-hostroot\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.139850 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.139663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-kubernetes\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.139850 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.139698 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv7hl\" (UniqueName: \"kubernetes.io/projected/7437f570-02f5-4131-8b0b-140ed029a668-kube-api-access-pv7hl\") pod \"node-ca-tvrzn\" (UID: \"7437f570-02f5-4131-8b0b-140ed029a668\") " pod="openshift-image-registry/node-ca-tvrzn" Oct 30 10:44:32.139850 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.139723 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-sysconfig\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.139850 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.139732 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqjpq\" (UniqueName: \"kubernetes.io/projected/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-kube-api-access-wqjpq\") pod \"network-metrics-daemon-4cgfp\" (UID: \"b06a8e3b-6db3-4cc7-b7e2-dc14122245ce\") " pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:32.139850 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.139763 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/275105df-bf86-436c-a464-890b4510c09d-system-cni-dir\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:44:32.140376 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.139778 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-var-lib-kubelet\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.140376 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.139811 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-kubelet\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.141118 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.141090 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-systemd-units\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.143653 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.143007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63196a40-0beb-4a8b-b396-0a78f57560f3-ovnkube-config\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.143653 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.143046 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c64ab741-bc4f-4006-932a-214cf53b6dc3-hosts-file\") pod \"node-resolver-gll67\" (UID: \"c64ab741-bc4f-4006-932a-214cf53b6dc3\") " pod="openshift-dns/node-resolver-gll67" Oct 30 10:44:32.143653 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.140492 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/538de0b2-f736-4723-92d6-e600b9de4a26-multus-daemon-config\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.143653 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.143079 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/45d9d286-4a6a-4ebb-91ab-324335c4a2ab-iptables-alerter-script\") pod \"iptables-alerter-vd8gp\" (UID: \"45d9d286-4a6a-4ebb-91ab-324335c4a2ab\") " pod="openshift-network-operator/iptables-alerter-vd8gp" Oct 30 10:44:32.143653 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.140699 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-kubernetes\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.143653 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.140778 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 10:44:32.143653 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.140675 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63196a40-0beb-4a8b-b396-0a78f57560f3-env-overrides\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.143653 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.140892 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-run-ovn-kubernetes\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.143653 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.140651 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-socket-dir\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:44:32.143653 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.140299 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/538de0b2-f736-4723-92d6-e600b9de4a26-cni-binary-copy\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.143653 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.141131 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-kubelet\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.143653 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.143267 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-host\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.143653 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.141172 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-systemd-units\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.143653 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:32.140652 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 10:44:32.143653 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.143300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-run\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.143653 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.143339 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-sys-fs\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:44:32.143653 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:32.143354 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs podName:b06a8e3b-6db3-4cc7-b7e2-dc14122245ce nodeName:}" failed. No retries permitted until 2026-10-30 10:44:32.643332118 +0000 UTC m=+3.065347403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs") pod "network-metrics-daemon-4cgfp" (UID: "b06a8e3b-6db3-4cc7-b7e2-dc14122245ce") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 10:44:32.143653 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.143398 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9-agent-certs\") pod \"konnectivity-agent-8xmhg\" (UID: \"fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9\") " pod="kube-system/konnectivity-agent-8xmhg" Oct 30 10:44:32.144632 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.143405 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-sys-fs\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:44:32.144632 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.141282 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63196a40-0beb-4a8b-b396-0a78f57560f3-ovnkube-script-lib\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.144632 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.143435 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c64ab741-bc4f-4006-932a-214cf53b6dc3-hosts-file\") pod \"node-resolver-gll67\" (UID: \"c64ab741-bc4f-4006-932a-214cf53b6dc3\") " pod="openshift-dns/node-resolver-gll67" Oct 30 10:44:32.144632 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.143487 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-host\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.144632 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.140843 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/275105df-bf86-436c-a464-890b4510c09d-system-cni-dir\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:44:32.144632 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.143488 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-run\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.144632 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.141110 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-lib-modules\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.144632 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.143720 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-cnibin\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.144632 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144103 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63196a40-0beb-4a8b-b396-0a78f57560f3-ovnkube-config\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.144632 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144156 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-cnibin\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.144632 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144194 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-run-netns\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.144632 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144221 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/45d9d286-4a6a-4ebb-91ab-324335c4a2ab-iptables-alerter-script\") pod \"iptables-alerter-vd8gp\" (UID: \"45d9d286-4a6a-4ebb-91ab-324335c4a2ab\") " pod="openshift-network-operator/iptables-alerter-vd8gp" Oct 30 10:44:32.144632 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144255 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-run-netns\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.144632 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144281 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7437f570-02f5-4131-8b0b-140ed029a668-host\") pod \"node-ca-tvrzn\" (UID: \"7437f570-02f5-4131-8b0b-140ed029a668\") " pod="openshift-image-registry/node-ca-tvrzn" Oct 30 10:44:32.144632 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144329 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7437f570-02f5-4131-8b0b-140ed029a668-serviceca\") pod \"node-ca-tvrzn\" (UID: \"7437f570-02f5-4131-8b0b-140ed029a668\") " pod="openshift-image-registry/node-ca-tvrzn" Oct 30 10:44:32.144632 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144367 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/275105df-bf86-436c-a464-890b4510c09d-cni-binary-copy\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:44:32.144632 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144391 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7437f570-02f5-4131-8b0b-140ed029a668-host\") pod \"node-ca-tvrzn\" (UID: \"7437f570-02f5-4131-8b0b-140ed029a668\") " pod="openshift-image-registry/node-ca-tvrzn" Oct 30 10:44:32.144632 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144419 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-var-lib-openvswitch\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.145554 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144499 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-var-lib-openvswitch\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.145554 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144545 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-log-socket\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.145554 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144574 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.145554 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144601 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8pl4\" (UniqueName: \"kubernetes.io/projected/45d9d286-4a6a-4ebb-91ab-324335c4a2ab-kube-api-access-q8pl4\") pod \"iptables-alerter-vd8gp\" (UID: \"45d9d286-4a6a-4ebb-91ab-324335c4a2ab\") " pod="openshift-network-operator/iptables-alerter-vd8gp" Oct 30 10:44:32.145554 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144622 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-var-lib-cni-bin\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.145554 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144641 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-registration-dir\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:44:32.145554 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144650 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-log-socket\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.145554 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144699 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-var-lib-cni-bin\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.145554 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144659 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-run-netns\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.145554 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144771 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-registration-dir\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:44:32.145554 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144826 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7437f570-02f5-4131-8b0b-140ed029a668-serviceca\") pod \"node-ca-tvrzn\" (UID: \"7437f570-02f5-4131-8b0b-140ed029a668\") " pod="openshift-image-registry/node-ca-tvrzn" Oct 30 10:44:32.145554 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144851 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.145554 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144920 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-run-openvswitch\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.145554 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144962 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-run-netns\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.145554 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.144997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-device-dir\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:44:32.145554 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145037 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-sys\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.145554 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145046 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-run-openvswitch\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.146114 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145022 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-device-dir\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:44:32.146114 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145079 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp4mk\" (UniqueName: \"kubernetes.io/projected/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8-kube-api-access-lp4mk\") pod \"network-check-target-fk5zl\" (UID: \"e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8\") " pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:44:32.146114 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145099 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-sys\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.146114 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145106 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9-konnectivity-ca\") pod \"konnectivity-agent-8xmhg\" (UID: \"fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9\") " pod="kube-system/konnectivity-agent-8xmhg" Oct 30 10:44:32.146114 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145039 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/275105df-bf86-436c-a464-890b4510c09d-cni-binary-copy\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:44:32.146114 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-multus-socket-dir-parent\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.146114 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145192 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-var-lib-cni-multus\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.146114 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145247 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-var-lib-cni-multus\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.146114 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145287 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrb2t\" (UniqueName: \"kubernetes.io/projected/538de0b2-f736-4723-92d6-e600b9de4a26-kube-api-access-xrb2t\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.146114 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145311 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-multus-socket-dir-parent\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.146114 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/275105df-bf86-436c-a464-890b4510c09d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:44:32.146114 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145395 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-run-systemd\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.146114 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58d4h\" (UniqueName: \"kubernetes.io/projected/63196a40-0beb-4a8b-b396-0a78f57560f3-kube-api-access-58d4h\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.146114 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145478 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-sysctl-d\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.146114 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/275105df-bf86-436c-a464-890b4510c09d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:44:32.146114 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzt9k\" (UniqueName: \"kubernetes.io/projected/275105df-bf86-436c-a464-890b4510c09d-kube-api-access-rzt9k\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:44:32.146114 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145585 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-node-log\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.146767 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145604 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-run-systemd\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.146767 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145621 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-cni-bin\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.146767 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145650 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4b288c6-5317-410f-9f88-0352f4164d20-etc-tuned\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.146767 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145670 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9-konnectivity-ca\") pod \"konnectivity-agent-8xmhg\" (UID: \"fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9\") " pod="kube-system/konnectivity-agent-8xmhg" Oct 30 10:44:32.146767 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145682 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-var-lib-kubelet\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.146767 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145780 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-node-log\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.146767 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145826 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-var-lib-kubelet\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.146767 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.145878 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-sysctl-d\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.146767 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.146237 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/275105df-bf86-436c-a464-890b4510c09d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:44:32.146767 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.146277 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-cni-bin\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.146767 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.146713 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/275105df-bf86-436c-a464-890b4510c09d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:44:32.148524 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.148479 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9-agent-certs\") pod \"konnectivity-agent-8xmhg\" (UID: \"fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9\") " pod="kube-system/konnectivity-agent-8xmhg" Oct 30 10:44:32.148673 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.148640 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4b288c6-5317-410f-9f88-0352f4164d20-etc-tuned\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.148752 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.148678 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4b288c6-5317-410f-9f88-0352f4164d20-tmp\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.150361 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.150338 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63196a40-0beb-4a8b-b396-0a78f57560f3-ovn-node-metrics-cert\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.170091 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.170029 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bflhl\" (UniqueName: \"kubernetes.io/projected/c64ab741-bc4f-4006-932a-214cf53b6dc3-kube-api-access-bflhl\") pod \"node-resolver-gll67\" (UID: \"c64ab741-bc4f-4006-932a-214cf53b6dc3\") " pod="openshift-dns/node-resolver-gll67" Oct 30 10:44:32.170200 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.170160 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv7hl\" (UniqueName: \"kubernetes.io/projected/7437f570-02f5-4131-8b0b-140ed029a668-kube-api-access-pv7hl\") pod \"node-ca-tvrzn\" (UID: \"7437f570-02f5-4131-8b0b-140ed029a668\") " pod="openshift-image-registry/node-ca-tvrzn" Oct 30 10:44:32.170270 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.170253 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqjpq\" (UniqueName: \"kubernetes.io/projected/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-kube-api-access-wqjpq\") pod \"network-metrics-daemon-4cgfp\" (UID: \"b06a8e3b-6db3-4cc7-b7e2-dc14122245ce\") " pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:32.170520 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:32.170373 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 10:44:32.170520 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:32.170395 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 10:44:32.170520 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:32.170408 2579 projected.go:196] Error preparing data for projected volume kube-api-access-lp4mk for pod openshift-network-diagnostics/network-check-target-fk5zl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 10:44:32.170520 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:32.170472 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8-kube-api-access-lp4mk podName:e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8 nodeName:}" failed. No retries permitted until 2026-10-30 10:44:32.670452794 +0000 UTC m=+3.092468100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lp4mk" (UniqueName: "kubernetes.io/projected/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8-kube-api-access-lp4mk") pod "network-check-target-fk5zl" (UID: "e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 10:44:32.171415 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.171384 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pzwb\" (UniqueName: \"kubernetes.io/projected/9f26208e-e65a-483d-ba73-d45fe31084bc-kube-api-access-4pzwb\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:44:32.171746 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.171723 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9g8f\" (UniqueName: \"kubernetes.io/projected/c4b288c6-5317-410f-9f88-0352f4164d20-kube-api-access-k9g8f\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.173229 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.173194 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrb2t\" (UniqueName: \"kubernetes.io/projected/538de0b2-f736-4723-92d6-e600b9de4a26-kube-api-access-xrb2t\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.173329 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.173196 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8pl4\" (UniqueName: \"kubernetes.io/projected/45d9d286-4a6a-4ebb-91ab-324335c4a2ab-kube-api-access-q8pl4\") pod \"iptables-alerter-vd8gp\" (UID: \"45d9d286-4a6a-4ebb-91ab-324335c4a2ab\") " pod="openshift-network-operator/iptables-alerter-vd8gp" Oct 30 10:44:32.173398 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.173341 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58d4h\" (UniqueName: \"kubernetes.io/projected/63196a40-0beb-4a8b-b396-0a78f57560f3-kube-api-access-58d4h\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.175089 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.175066 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzt9k\" (UniqueName: \"kubernetes.io/projected/275105df-bf86-436c-a464-890b4510c09d-kube-api-access-rzt9k\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:44:32.253164 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.253124 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vd8gp" Oct 30 10:44:32.265785 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:32.265754 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d9d286_4a6a_4ebb_91ab_324335c4a2ab.slice/crio-545d644eb3c1bb16fdd8358db707739b2bb7321f97b3f5e75edadc84ba11168f WatchSource:0}: Error finding container 545d644eb3c1bb16fdd8358db707739b2bb7321f97b3f5e75edadc84ba11168f: Status 404 returned error can't find the container with id 545d644eb3c1bb16fdd8358db707739b2bb7321f97b3f5e75edadc84ba11168f Oct 30 10:44:32.289050 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.289018 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:44:32.292162 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.292132 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:44:32.293394 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.293360 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:44:32.296118 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.296096 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8xmhg" Oct 30 10:44:32.301926 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.301907 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ftv4s" Oct 30 10:44:32.302576 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:32.302548 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275105df_bf86_436c_a464_890b4510c09d.slice/crio-1389d1ff3c310f1cf00b31a6d1be34fbf370e4b033f5285d965c957418f67274 WatchSource:0}: Error finding container 1389d1ff3c310f1cf00b31a6d1be34fbf370e4b033f5285d965c957418f67274: Status 404 returned error can't find the container with id 1389d1ff3c310f1cf00b31a6d1be34fbf370e4b033f5285d965c957418f67274 Oct 30 10:44:32.303829 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:32.303804 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b288c6_5317_410f_9f88_0352f4164d20.slice/crio-5e233668e74f0b4a8a940d641dddfc6b0d9b1bd4caa94bbd53a3ba59e7e00cd5 WatchSource:0}: Error finding container 5e233668e74f0b4a8a940d641dddfc6b0d9b1bd4caa94bbd53a3ba59e7e00cd5: Status 404 returned error can't find the container with id 5e233668e74f0b4a8a940d641dddfc6b0d9b1bd4caa94bbd53a3ba59e7e00cd5 Oct 30 10:44:32.311540 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:32.311483 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f26208e_e65a_483d_ba73_d45fe31084bc.slice/crio-48a8eb8a2666149b54b9c3136708009c599aba6caabd13677a5dfad1e0311aac WatchSource:0}: Error finding container 48a8eb8a2666149b54b9c3136708009c599aba6caabd13677a5dfad1e0311aac: Status 404 returned error can't find the container with id 48a8eb8a2666149b54b9c3136708009c599aba6caabd13677a5dfad1e0311aac Oct 30 10:44:32.312491 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:32.312465 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6114a1_4640_4ede_86a2_d6bb4aa4e2d9.slice/crio-4c5a6238e5c42ce712e1c576150adf2b35ce71c93d62fddc1a72c8a6a7b515ee WatchSource:0}: Error finding container 4c5a6238e5c42ce712e1c576150adf2b35ce71c93d62fddc1a72c8a6a7b515ee: Status 404 returned error can't find the container with id 4c5a6238e5c42ce712e1c576150adf2b35ce71c93d62fddc1a72c8a6a7b515ee Oct 30 10:44:32.317210 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:32.317174 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod538de0b2_f736_4723_92d6_e600b9de4a26.slice/crio-f2d110367251c32e839f6d0a99742af4fec95018d18f554439b4b7a9f54a920b WatchSource:0}: Error finding container f2d110367251c32e839f6d0a99742af4fec95018d18f554439b4b7a9f54a920b: Status 404 returned error can't find the container with id f2d110367251c32e839f6d0a99742af4fec95018d18f554439b4b7a9f54a920b Oct 30 10:44:32.327760 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.327733 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tvrzn" Oct 30 10:44:32.329164 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.329135 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gll67" Oct 30 10:44:32.331946 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.331905 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:32.339557 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:32.339531 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7437f570_02f5_4131_8b0b_140ed029a668.slice/crio-fc410460dda0dc3b4b216b489316da2ebc81a3aee74e7461260ed8ae49db6e94 WatchSource:0}: Error finding container fc410460dda0dc3b4b216b489316da2ebc81a3aee74e7461260ed8ae49db6e94: Status 404 returned error can't find the container with id fc410460dda0dc3b4b216b489316da2ebc81a3aee74e7461260ed8ae49db6e94 Oct 30 10:44:32.342949 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:32.342910 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc64ab741_bc4f_4006_932a_214cf53b6dc3.slice/crio-149f994b0fde7bb8a2e011e60cf13a1aa60718df8878ae2891567ce7dae652c6 WatchSource:0}: Error finding container 149f994b0fde7bb8a2e011e60cf13a1aa60718df8878ae2891567ce7dae652c6: Status 404 returned error can't find the container with id 149f994b0fde7bb8a2e011e60cf13a1aa60718df8878ae2891567ce7dae652c6 Oct 30 10:44:32.346476 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:44:32.346452 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63196a40_0beb_4a8b_b396_0a78f57560f3.slice/crio-00c3d5844216d347f4d1b516b3a43e4b86b7fa9b97a7a2b78bf8b6ca66c0945a WatchSource:0}: Error finding container 00c3d5844216d347f4d1b516b3a43e4b86b7fa9b97a7a2b78bf8b6ca66c0945a: Status 404 returned error can't find the container with id 00c3d5844216d347f4d1b516b3a43e4b86b7fa9b97a7a2b78bf8b6ca66c0945a Oct 30 10:44:32.377666 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.377630 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 10:44:32.650545 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.650510 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs\") pod \"network-metrics-daemon-4cgfp\" (UID: \"b06a8e3b-6db3-4cc7-b7e2-dc14122245ce\") " pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:32.650742 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.650603 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/23abd71b-1521-4aec-9d79-198f0f8e14a2-original-pull-secret\") pod \"global-pull-secret-syncer-vskj5\" (UID: \"23abd71b-1521-4aec-9d79-198f0f8e14a2\") " pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:32.650742 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:32.650644 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 10:44:32.650742 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:32.650701 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 10:44:32.650742 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:32.650729 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs podName:b06a8e3b-6db3-4cc7-b7e2-dc14122245ce nodeName:}" failed. No retries permitted until 2026-10-30 10:44:33.650709215 +0000 UTC m=+4.072724510 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs") pod "network-metrics-daemon-4cgfp" (UID: "b06a8e3b-6db3-4cc7-b7e2-dc14122245ce") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 10:44:32.650960 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:32.650752 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23abd71b-1521-4aec-9d79-198f0f8e14a2-original-pull-secret podName:23abd71b-1521-4aec-9d79-198f0f8e14a2 nodeName:}" failed. No retries permitted until 2026-10-30 10:44:33.650741444 +0000 UTC m=+4.072756725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/23abd71b-1521-4aec-9d79-198f0f8e14a2-original-pull-secret") pod "global-pull-secret-syncer-vskj5" (UID: "23abd71b-1521-4aec-9d79-198f0f8e14a2") : object "kube-system"/"original-pull-secret" not registered Oct 30 10:44:32.752199 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.752047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp4mk\" (UniqueName: \"kubernetes.io/projected/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8-kube-api-access-lp4mk\") pod \"network-check-target-fk5zl\" (UID: \"e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8\") " pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:44:32.752334 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:32.752238 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 10:44:32.752334 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:32.752264 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 10:44:32.752334 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:32.752279 2579 projected.go:196] Error preparing data for projected volume kube-api-access-lp4mk for pod openshift-network-diagnostics/network-check-target-fk5zl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 10:44:32.752334 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:32.752335 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8-kube-api-access-lp4mk podName:e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8 nodeName:}" failed. No retries permitted until 2026-10-30 10:44:33.752315095 +0000 UTC m=+4.174330393 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lp4mk" (UniqueName: "kubernetes.io/projected/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8-kube-api-access-lp4mk") pod "network-check-target-fk5zl" (UID: "e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 10:44:32.990560 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.990211 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 10:39:32 +0000 UTC" deadline="2028-07-07 16:27:33.896282298 +0000 UTC" Oct 30 10:44:32.990560 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:32.990501 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14789h43m0.905787092s" Oct 30 10:44:33.077079 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:33.076894 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vd8gp" event={"ID":"45d9d286-4a6a-4ebb-91ab-324335c4a2ab","Type":"ContainerStarted","Data":"545d644eb3c1bb16fdd8358db707739b2bb7321f97b3f5e75edadc84ba11168f"} Oct 30 10:44:33.079595 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:33.079452 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" event={"ID":"63196a40-0beb-4a8b-b396-0a78f57560f3","Type":"ContainerStarted","Data":"00c3d5844216d347f4d1b516b3a43e4b86b7fa9b97a7a2b78bf8b6ca66c0945a"} Oct 30 10:44:33.086629 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:33.086554 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tvrzn" event={"ID":"7437f570-02f5-4131-8b0b-140ed029a668","Type":"ContainerStarted","Data":"fc410460dda0dc3b4b216b489316da2ebc81a3aee74e7461260ed8ae49db6e94"} Oct 30 10:44:33.099984 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:33.099916 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8xmhg" event={"ID":"fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9","Type":"ContainerStarted","Data":"4c5a6238e5c42ce712e1c576150adf2b35ce71c93d62fddc1a72c8a6a7b515ee"} Oct 30 10:44:33.102543 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:33.102467 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" event={"ID":"9f26208e-e65a-483d-ba73-d45fe31084bc","Type":"ContainerStarted","Data":"48a8eb8a2666149b54b9c3136708009c599aba6caabd13677a5dfad1e0311aac"} Oct 30 10:44:33.111728 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:33.111645 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98jtw" event={"ID":"275105df-bf86-436c-a464-890b4510c09d","Type":"ContainerStarted","Data":"1389d1ff3c310f1cf00b31a6d1be34fbf370e4b033f5285d965c957418f67274"} Oct 30 10:44:33.120281 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:33.119989 2579 generic.go:356] "Generic (PLEG): container finished" podID="d39640f4fcc35209a29f428a929973b1" containerID="7a52ac7f1460d3889e0c6831de7897cf7a04484fa3fb234174a09a536692a6c7" exitCode=0 Oct 30 10:44:33.120281 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:33.120088 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" event={"ID":"d39640f4fcc35209a29f428a929973b1","Type":"ContainerDied","Data":"7a52ac7f1460d3889e0c6831de7897cf7a04484fa3fb234174a09a536692a6c7"} Oct 30 10:44:33.130121 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:33.130038 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" event={"ID":"6dc9a86d1bbc0350320467df988d567a","Type":"ContainerStarted","Data":"5841a65763d434d3306679f75b96c7b82719f99081c043a638431211ebb7c16d"} Oct 30 10:44:33.133430 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:33.133356 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gll67" event={"ID":"c64ab741-bc4f-4006-932a-214cf53b6dc3","Type":"ContainerStarted","Data":"149f994b0fde7bb8a2e011e60cf13a1aa60718df8878ae2891567ce7dae652c6"} Oct 30 10:44:33.149924 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:33.149893 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ftv4s" event={"ID":"538de0b2-f736-4723-92d6-e600b9de4a26","Type":"ContainerStarted","Data":"f2d110367251c32e839f6d0a99742af4fec95018d18f554439b4b7a9f54a920b"} Oct 30 10:44:33.155393 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:33.155364 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" event={"ID":"c4b288c6-5317-410f-9f88-0352f4164d20","Type":"ContainerStarted","Data":"5e233668e74f0b4a8a940d641dddfc6b0d9b1bd4caa94bbd53a3ba59e7e00cd5"} Oct 30 10:44:33.658807 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:33.658717 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs\") pod \"network-metrics-daemon-4cgfp\" (UID: \"b06a8e3b-6db3-4cc7-b7e2-dc14122245ce\") " pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:33.658807 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:33.658783 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/23abd71b-1521-4aec-9d79-198f0f8e14a2-original-pull-secret\") pod \"global-pull-secret-syncer-vskj5\" (UID: \"23abd71b-1521-4aec-9d79-198f0f8e14a2\") " pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:33.659037 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:33.658872 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 10:44:33.659037 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:33.658915 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23abd71b-1521-4aec-9d79-198f0f8e14a2-original-pull-secret podName:23abd71b-1521-4aec-9d79-198f0f8e14a2 nodeName:}" failed. No retries permitted until 2026-10-30 10:44:35.658903478 +0000 UTC m=+6.080918759 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/23abd71b-1521-4aec-9d79-198f0f8e14a2-original-pull-secret") pod "global-pull-secret-syncer-vskj5" (UID: "23abd71b-1521-4aec-9d79-198f0f8e14a2") : object "kube-system"/"original-pull-secret" not registered Oct 30 10:44:33.659288 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:33.659270 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 10:44:33.659393 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:33.659318 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs podName:b06a8e3b-6db3-4cc7-b7e2-dc14122245ce nodeName:}" failed. No retries permitted until 2026-10-30 10:44:35.659299584 +0000 UTC m=+6.081314878 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs") pod "network-metrics-daemon-4cgfp" (UID: "b06a8e3b-6db3-4cc7-b7e2-dc14122245ce") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 10:44:33.760513 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:33.759813 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp4mk\" (UniqueName: \"kubernetes.io/projected/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8-kube-api-access-lp4mk\") pod \"network-check-target-fk5zl\" (UID: \"e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8\") " pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:44:33.760513 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:33.760049 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 10:44:33.760513 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:33.760073 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 10:44:33.760513 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:33.760087 2579 projected.go:196] Error preparing data for projected volume kube-api-access-lp4mk for pod openshift-network-diagnostics/network-check-target-fk5zl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 10:44:33.760513 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:33.760148 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8-kube-api-access-lp4mk podName:e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8 nodeName:}" failed. No retries permitted until 2026-10-30 10:44:35.760128266 +0000 UTC m=+6.182143562 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lp4mk" (UniqueName: "kubernetes.io/projected/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8-kube-api-access-lp4mk") pod "network-check-target-fk5zl" (UID: "e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 10:44:33.991082 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:33.990997 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 10:39:32 +0000 UTC" deadline="2028-06-05 02:06:21.980552546 +0000 UTC" Oct 30 10:44:33.991082 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:33.991040 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14007h21m47.989516859s" Oct 30 10:44:34.046846 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:34.046813 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:44:34.046999 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:34.046951 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fk5zl" podUID="e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8" Oct 30 10:44:34.047104 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:34.047085 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:34.047209 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:34.047186 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vskj5" podUID="23abd71b-1521-4aec-9d79-198f0f8e14a2" Oct 30 10:44:34.047293 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:34.047270 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:34.047393 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:34.047374 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cgfp" podUID="b06a8e3b-6db3-4cc7-b7e2-dc14122245ce" Oct 30 10:44:34.175916 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:34.175822 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" event={"ID":"d39640f4fcc35209a29f428a929973b1","Type":"ContainerStarted","Data":"d7a56cdbf1cdce5c93a90066a6bc40360249c0aa95ff2a2402bab2a15a1ab669"} Oct 30 10:44:34.221875 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:34.221804 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" podStartSLOduration=3.221786837 podStartE2EDuration="3.221786837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 10:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:44:34.221643794 +0000 UTC m=+4.643659098" watchObservedRunningTime="2026-10-30 10:44:34.221786837 +0000 UTC m=+4.643802142" Oct 30 10:44:34.222158 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:34.222124 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" podStartSLOduration=3.222114439 podStartE2EDuration="3.222114439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 10:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:44:33.208112808 +0000 UTC m=+3.630128113" watchObservedRunningTime="2026-10-30 10:44:34.222114439 +0000 UTC m=+4.644129744" Oct 30 10:44:35.676927 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:35.676830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/23abd71b-1521-4aec-9d79-198f0f8e14a2-original-pull-secret\") pod \"global-pull-secret-syncer-vskj5\" (UID: \"23abd71b-1521-4aec-9d79-198f0f8e14a2\") " pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:35.676927 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:35.676890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs\") pod \"network-metrics-daemon-4cgfp\" (UID: \"b06a8e3b-6db3-4cc7-b7e2-dc14122245ce\") " pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:35.677501 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:35.677051 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 10:44:35.677501 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:35.677113 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs podName:b06a8e3b-6db3-4cc7-b7e2-dc14122245ce nodeName:}" failed. No retries permitted until 2026-10-30 10:44:39.677095063 +0000 UTC m=+10.099110345 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs") pod "network-metrics-daemon-4cgfp" (UID: "b06a8e3b-6db3-4cc7-b7e2-dc14122245ce") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 10:44:35.677638 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:35.677557 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 10:44:35.677638 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:35.677604 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23abd71b-1521-4aec-9d79-198f0f8e14a2-original-pull-secret podName:23abd71b-1521-4aec-9d79-198f0f8e14a2 nodeName:}" failed. No retries permitted until 2026-10-30 10:44:39.677588665 +0000 UTC m=+10.099603948 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/23abd71b-1521-4aec-9d79-198f0f8e14a2-original-pull-secret") pod "global-pull-secret-syncer-vskj5" (UID: "23abd71b-1521-4aec-9d79-198f0f8e14a2") : object "kube-system"/"original-pull-secret" not registered Oct 30 10:44:35.777898 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:35.777855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp4mk\" (UniqueName: \"kubernetes.io/projected/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8-kube-api-access-lp4mk\") pod \"network-check-target-fk5zl\" (UID: \"e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8\") " pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:44:35.778075 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:35.778053 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 10:44:35.778075 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:35.778073 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 10:44:35.778287 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:35.778086 2579 projected.go:196] Error preparing data for projected volume kube-api-access-lp4mk for pod openshift-network-diagnostics/network-check-target-fk5zl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 10:44:35.778287 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:35.778143 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8-kube-api-access-lp4mk podName:e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8 nodeName:}" failed. No retries permitted until 2026-10-30 10:44:39.778125113 +0000 UTC m=+10.200140418 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lp4mk" (UniqueName: "kubernetes.io/projected/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8-kube-api-access-lp4mk") pod "network-check-target-fk5zl" (UID: "e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 10:44:36.048270 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:36.047340 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:36.048270 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:36.047496 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cgfp" podUID="b06a8e3b-6db3-4cc7-b7e2-dc14122245ce" Oct 30 10:44:36.048270 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:36.047955 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:44:36.048270 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:36.048048 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fk5zl" podUID="e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8" Oct 30 10:44:36.048270 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:36.048120 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:36.048270 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:36.048202 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vskj5" podUID="23abd71b-1521-4aec-9d79-198f0f8e14a2" Oct 30 10:44:36.189083 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:36.188147 2579 generic.go:356] "Generic (PLEG): container finished" podID="275105df-bf86-436c-a464-890b4510c09d" containerID="93ca804e6a9a413976d9635ca8677cba17f9481052c90e4c3b61f6beefbd5245" exitCode=0 Oct 30 10:44:36.189083 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:36.188207 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98jtw" event={"ID":"275105df-bf86-436c-a464-890b4510c09d","Type":"ContainerDied","Data":"93ca804e6a9a413976d9635ca8677cba17f9481052c90e4c3b61f6beefbd5245"} Oct 30 10:44:37.194655 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:37.194615 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8xmhg" event={"ID":"fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9","Type":"ContainerStarted","Data":"f57c0b0638ae991ab4897ffce7d5da813b49c30860f9d8578f88abf193e53b54"} Oct 30 10:44:37.237347 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:37.237220 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8xmhg" podStartSLOduration=3.100908628 podStartE2EDuration="7.237198428s" totalImagesPullingTime="4.1362898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:44:37.236201146 +0000 UTC m=+7.658216447" watchObservedRunningTime="2026-10-30 10:44:37.237198428 +0000 UTC m=+7.659213746" Oct 30 10:44:38.045347 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:38.045318 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:38.045347 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:38.045339 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:44:38.045550 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:38.045318 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:38.045550 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:38.045464 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vskj5" podUID="23abd71b-1521-4aec-9d79-198f0f8e14a2" Oct 30 10:44:38.045653 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:38.045629 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cgfp" podUID="b06a8e3b-6db3-4cc7-b7e2-dc14122245ce" Oct 30 10:44:38.045715 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:38.045698 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fk5zl" podUID="e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8" Oct 30 10:44:38.201408 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:38.201370 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tvrzn" event={"ID":"7437f570-02f5-4131-8b0b-140ed029a668","Type":"ContainerStarted","Data":"70b697c3e4850f72ce35696d97694e4856de6de76d2578985a76b9a2fcd488dc"} Oct 30 10:44:38.242045 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:38.241986 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tvrzn" podStartSLOduration=3.133749924 podStartE2EDuration="8.241968897s" totalImagesPullingTime="5.108218973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 10:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:44:38.241413141 +0000 UTC m=+8.663428442" watchObservedRunningTime="2026-10-30 10:44:38.241968897 +0000 UTC m=+8.663984201" Oct 30 10:44:39.208122 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:39.208080 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" event={"ID":"9f26208e-e65a-483d-ba73-d45fe31084bc","Type":"ContainerStarted","Data":"470699b60f06c64ef699282f50706e904a0de6d96ca16203ec14937c282d4150"} Oct 30 10:44:39.584538 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:39.584507 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 10:44:39.711006 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:39.710964 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs\") pod \"network-metrics-daemon-4cgfp\" (UID: \"b06a8e3b-6db3-4cc7-b7e2-dc14122245ce\") " pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:39.711165 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:39.711063 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/23abd71b-1521-4aec-9d79-198f0f8e14a2-original-pull-secret\") pod \"global-pull-secret-syncer-vskj5\" (UID: \"23abd71b-1521-4aec-9d79-198f0f8e14a2\") " pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:39.711236 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:39.711180 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 10:44:39.711300 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:39.711244 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23abd71b-1521-4aec-9d79-198f0f8e14a2-original-pull-secret podName:23abd71b-1521-4aec-9d79-198f0f8e14a2 nodeName:}" failed. No retries permitted until 2026-10-30 10:44:47.711226659 +0000 UTC m=+18.133241941 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/23abd71b-1521-4aec-9d79-198f0f8e14a2-original-pull-secret") pod "global-pull-secret-syncer-vskj5" (UID: "23abd71b-1521-4aec-9d79-198f0f8e14a2") : object "kube-system"/"original-pull-secret" not registered Oct 30 10:44:39.711363 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:39.711313 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 10:44:39.711363 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:39.711345 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs podName:b06a8e3b-6db3-4cc7-b7e2-dc14122245ce nodeName:}" failed. No retries permitted until 2026-10-30 10:44:47.711334221 +0000 UTC m=+18.133349501 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs") pod "network-metrics-daemon-4cgfp" (UID: "b06a8e3b-6db3-4cc7-b7e2-dc14122245ce") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 10:44:39.811993 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:39.811912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp4mk\" (UniqueName: \"kubernetes.io/projected/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8-kube-api-access-lp4mk\") pod \"network-check-target-fk5zl\" (UID: \"e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8\") " pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:44:39.812259 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:39.812233 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 10:44:39.812350 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:39.812263 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 10:44:39.812350 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:39.812277 2579 projected.go:196] Error preparing data for projected volume kube-api-access-lp4mk for pod openshift-network-diagnostics/network-check-target-fk5zl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 10:44:39.812350 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:39.812336 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8-kube-api-access-lp4mk podName:e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8 nodeName:}" failed. No retries permitted until 2026-10-30 10:44:47.812317476 +0000 UTC m=+18.234332778 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lp4mk" (UniqueName: "kubernetes.io/projected/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8-kube-api-access-lp4mk") pod "network-check-target-fk5zl" (UID: "e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 10:44:39.989823 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:39.989750 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T10:44:39.584526522Z","UUID":"a9ec8ff5-9b11-4237-be8a-28105755ab86","Handler":null,"Name":"","Endpoint":""} Oct 30 10:44:39.994138 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:39.994114 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 10:44:39.994249 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:39.994148 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 10:44:40.045160 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:40.045131 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:40.045482 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:40.045461 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:40.045585 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:40.045568 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cgfp" podUID="b06a8e3b-6db3-4cc7-b7e2-dc14122245ce" Oct 30 10:44:40.045651 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:40.045628 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:44:40.045704 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:40.045658 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vskj5" podUID="23abd71b-1521-4aec-9d79-198f0f8e14a2" Oct 30 10:44:40.045756 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:40.045729 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fk5zl" podUID="e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8" Oct 30 10:44:40.217247 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:40.217020 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ftv4s" event={"ID":"538de0b2-f736-4723-92d6-e600b9de4a26","Type":"ContainerStarted","Data":"51a6f18ceba8624d29970edf0b39f493edb21be547bac46f8ccc72e9069b1330"} Oct 30 10:44:40.221656 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:40.221623 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" event={"ID":"9f26208e-e65a-483d-ba73-d45fe31084bc","Type":"ContainerStarted","Data":"d2be950835b4d86d390c8edec89bbc4672412e141829d341bcf42a67f264a1a3"} Oct 30 10:44:40.261449 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:40.261380 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-ftv4s" podStartSLOduration=2.62755032 podStartE2EDuration="10.261361396s" totalImagesPullingTime="7.633811076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 10:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:44:40.260737542 +0000 UTC m=+10.682752842" watchObservedRunningTime="2026-10-30 10:44:40.261361396 +0000 UTC m=+10.683376696" Oct 30 10:44:40.931070 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:40.930731 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8xmhg" Oct 30 10:44:40.931471 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:40.931442 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8xmhg" Oct 30 10:44:41.226531 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:41.226492 2579 generic.go:356] "Generic (PLEG): container finished" podID="275105df-bf86-436c-a464-890b4510c09d" containerID="06e4d993055de8f1b26a5f90b2903f29091c41b4d13228ae52fcdd8612d52db5" exitCode=0 Oct 30 10:44:41.227006 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:41.226588 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98jtw" event={"ID":"275105df-bf86-436c-a464-890b4510c09d","Type":"ContainerDied","Data":"06e4d993055de8f1b26a5f90b2903f29091c41b4d13228ae52fcdd8612d52db5"} Oct 30 10:44:41.230778 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:41.230756 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gll67" event={"ID":"c64ab741-bc4f-4006-932a-214cf53b6dc3","Type":"ContainerStarted","Data":"a92102037ac6e139b87cf2dde6e88be115dbdc0b2e826c6ef1120164c2f4d461"} Oct 30 10:44:41.234059 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:41.234036 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" event={"ID":"9f26208e-e65a-483d-ba73-d45fe31084bc","Type":"ContainerStarted","Data":"7b5f06983e5488ff2384792bece3a67508793b65737fd5f5f5ea7f418b1b5db4"} Oct 30 10:44:41.317965 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:41.317907 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" podStartSLOduration=3.222339716 podStartE2EDuration="11.317888952s" totalImagesPullingTime="8.095549236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 10:44:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:44:41.317840673 +0000 UTC m=+11.739855974" watchObservedRunningTime="2026-10-30 10:44:41.317888952 +0000 UTC m=+11.739904247" Oct 30 10:44:41.358386 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:41.358326 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-gll67" podStartSLOduration=3.369597455 podStartE2EDuration="11.358309649s" totalImagesPullingTime="7.988712194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 10:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:44:41.358242986 +0000 UTC m=+11.780258286" watchObservedRunningTime="2026-10-30 10:44:41.358309649 +0000 UTC m=+11.780324957" Oct 30 10:44:42.044406 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:42.044374 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:42.044406 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:42.044407 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:44:42.044614 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:42.044373 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:42.044614 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:42.044514 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fk5zl" podUID="e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8" Oct 30 10:44:42.045146 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:42.045117 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cgfp" podUID="b06a8e3b-6db3-4cc7-b7e2-dc14122245ce" Oct 30 10:44:42.045255 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:42.045232 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vskj5" podUID="23abd71b-1521-4aec-9d79-198f0f8e14a2" Oct 30 10:44:42.237098 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:42.237066 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" event={"ID":"c4b288c6-5317-410f-9f88-0352f4164d20","Type":"ContainerStarted","Data":"44c68c105ad375bce6bc8c5dfdc995e693d112f10df992ffda656372dbea19f6"} Oct 30 10:44:42.239623 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:42.239597 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vd8gp" event={"ID":"45d9d286-4a6a-4ebb-91ab-324335c4a2ab","Type":"ContainerStarted","Data":"994820bac260b729a7f41445ec99160dc6fbb2cc052d627aa1ea9d420b41b7a4"} Oct 30 10:44:42.243582 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:42.243475 2579 generic.go:356] "Generic (PLEG): container finished" podID="275105df-bf86-436c-a464-890b4510c09d" containerID="afa6a9b7c1ea513bec91abaa8837d83f171246373b0fd63676c04f65a51854c2" exitCode=0 Oct 30 10:44:42.243671 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:42.243570 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98jtw" event={"ID":"275105df-bf86-436c-a464-890b4510c09d","Type":"ContainerDied","Data":"afa6a9b7c1ea513bec91abaa8837d83f171246373b0fd63676c04f65a51854c2"} Oct 30 10:44:42.279857 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:42.279796 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vd8gp" podStartSLOduration=4.060111158 podStartE2EDuration="12.279779722s" totalImagesPullingTime="8.219668564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 10:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:44:42.279607929 +0000 UTC m=+12.701623230" watchObservedRunningTime="2026-10-30 10:44:42.279779722 +0000 UTC m=+12.701795021" Oct 30 10:44:43.248112 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:43.248080 2579 generic.go:356] "Generic (PLEG): container finished" podID="275105df-bf86-436c-a464-890b4510c09d" containerID="86fda8e65f937ca9ff90b16ad9a7c6b59dbd8c5689d1c9723b032d306755af3d" exitCode=0 Oct 30 10:44:43.248701 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:43.248170 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98jtw" event={"ID":"275105df-bf86-436c-a464-890b4510c09d","Type":"ContainerDied","Data":"86fda8e65f937ca9ff90b16ad9a7c6b59dbd8c5689d1c9723b032d306755af3d"} Oct 30 10:44:43.366409 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:43.366357 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" podStartSLOduration=3.544680944 podStartE2EDuration="13.36634308s" totalImagesPullingTime="9.821662136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 10:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:44:43.365845322 +0000 UTC m=+13.787860622" watchObservedRunningTime="2026-10-30 10:44:43.36634308 +0000 UTC m=+13.788358380" Oct 30 10:44:44.044442 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:44.044392 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:44.044609 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:44.044542 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vskj5" podUID="23abd71b-1521-4aec-9d79-198f0f8e14a2" Oct 30 10:44:44.044973 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:44.044953 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:44.045085 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:44.045066 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cgfp" podUID="b06a8e3b-6db3-4cc7-b7e2-dc14122245ce" Oct 30 10:44:44.045247 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:44.045222 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:44:44.045341 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:44.045308 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fk5zl" podUID="e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8" Oct 30 10:44:45.256380 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:45.256158 2579 generic.go:356] "Generic (PLEG): container finished" podID="275105df-bf86-436c-a464-890b4510c09d" containerID="c1839d7b54097b6030dd266a28b7555b2e80d48128ac87b7a9e440e6b6b343e9" exitCode=0 Oct 30 10:44:45.256380 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:45.256204 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98jtw" event={"ID":"275105df-bf86-436c-a464-890b4510c09d","Type":"ContainerDied","Data":"c1839d7b54097b6030dd266a28b7555b2e80d48128ac87b7a9e440e6b6b343e9"} Oct 30 10:44:46.044516 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:46.044484 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:46.044670 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:46.044484 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:46.044670 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:46.044596 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vskj5" podUID="23abd71b-1521-4aec-9d79-198f0f8e14a2" Oct 30 10:44:46.044751 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:46.044695 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cgfp" podUID="b06a8e3b-6db3-4cc7-b7e2-dc14122245ce" Oct 30 10:44:46.044751 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:46.044498 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:44:46.044826 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:46.044782 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fk5zl" podUID="e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8" Oct 30 10:44:46.261941 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:46.261898 2579 generic.go:356] "Generic (PLEG): container finished" podID="275105df-bf86-436c-a464-890b4510c09d" containerID="bf3eb3226aa05ffebbd61a93c15bff17e05448727d3290ee5fe02f804254f304" exitCode=0 Oct 30 10:44:46.262475 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:46.261960 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98jtw" event={"ID":"275105df-bf86-436c-a464-890b4510c09d","Type":"ContainerDied","Data":"bf3eb3226aa05ffebbd61a93c15bff17e05448727d3290ee5fe02f804254f304"} Oct 30 10:44:47.266213 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:47.265999 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" event={"ID":"63196a40-0beb-4a8b-b396-0a78f57560f3","Type":"ContainerStarted","Data":"f221649a25101230a883416a9c7763a70c34ee51037efabeb1076a6424ecd263"} Oct 30 10:44:47.266624 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:47.266305 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" event={"ID":"63196a40-0beb-4a8b-b396-0a78f57560f3","Type":"ContainerStarted","Data":"272d5500ef3ec88cb910e2244f1486627f9499006e726fb7659d2af761d1bc21"} Oct 30 10:44:47.266624 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:47.266362 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" event={"ID":"63196a40-0beb-4a8b-b396-0a78f57560f3","Type":"ContainerStarted","Data":"d78b689aee4560512fbd4d8435a50a0f08c81506aa4a6b01d579c97081370cfe"} Oct 30 10:44:47.266624 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:47.266410 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" event={"ID":"63196a40-0beb-4a8b-b396-0a78f57560f3","Type":"ContainerStarted","Data":"dd6076d0f5b138a04eab49f05af7a6478442d750b81b86ff7996f4d500f31a8a"} Oct 30 10:44:47.269370 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:47.269351 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98jtw" event={"ID":"275105df-bf86-436c-a464-890b4510c09d","Type":"ContainerStarted","Data":"bcf76d048ef525bd2ca27cc9f801ffcef88087854e1790e232878e3823f80064"} Oct 30 10:44:47.344170 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:47.344122 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-98jtw" podStartSLOduration=7.215378682 podStartE2EDuration="17.344110044s" totalImagesPullingTime="10.128731362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 10:44:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:44:47.330376182 +0000 UTC m=+17.752391484" watchObservedRunningTime="2026-10-30 10:44:47.344110044 +0000 UTC m=+17.766125343" Oct 30 10:44:47.767642 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:47.767612 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/23abd71b-1521-4aec-9d79-198f0f8e14a2-original-pull-secret\") pod \"global-pull-secret-syncer-vskj5\" (UID: \"23abd71b-1521-4aec-9d79-198f0f8e14a2\") " pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:47.767835 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:47.767650 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs\") pod \"network-metrics-daemon-4cgfp\" (UID: \"b06a8e3b-6db3-4cc7-b7e2-dc14122245ce\") " pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:47.767835 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:47.767739 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 10:44:47.767835 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:47.767765 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 10:44:47.767835 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:47.767790 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs podName:b06a8e3b-6db3-4cc7-b7e2-dc14122245ce nodeName:}" failed. No retries permitted until 2026-10-30 10:45:03.767775232 +0000 UTC m=+34.189790511 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs") pod "network-metrics-daemon-4cgfp" (UID: "b06a8e3b-6db3-4cc7-b7e2-dc14122245ce") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 10:44:47.767835 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:47.767829 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23abd71b-1521-4aec-9d79-198f0f8e14a2-original-pull-secret podName:23abd71b-1521-4aec-9d79-198f0f8e14a2 nodeName:}" failed. No retries permitted until 2026-10-30 10:45:03.767813367 +0000 UTC m=+34.189828646 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/23abd71b-1521-4aec-9d79-198f0f8e14a2-original-pull-secret") pod "global-pull-secret-syncer-vskj5" (UID: "23abd71b-1521-4aec-9d79-198f0f8e14a2") : object "kube-system"/"original-pull-secret" not registered Oct 30 10:44:47.868833 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:47.868794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp4mk\" (UniqueName: \"kubernetes.io/projected/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8-kube-api-access-lp4mk\") pod \"network-check-target-fk5zl\" (UID: \"e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8\") " pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:44:47.868974 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:47.868922 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 10:44:47.868974 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:47.868937 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 10:44:47.868974 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:47.868945 2579 projected.go:196] Error preparing data for projected volume kube-api-access-lp4mk for pod openshift-network-diagnostics/network-check-target-fk5zl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 10:44:47.869088 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:47.868991 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8-kube-api-access-lp4mk podName:e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8 nodeName:}" failed. No retries permitted until 2026-10-30 10:45:03.868978053 +0000 UTC m=+34.290993331 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lp4mk" (UniqueName: "kubernetes.io/projected/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8-kube-api-access-lp4mk") pod "network-check-target-fk5zl" (UID: "e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 10:44:48.044205 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:48.044123 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:48.044205 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:48.044174 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:48.044205 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:48.044188 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:44:48.044381 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:48.044268 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cgfp" podUID="b06a8e3b-6db3-4cc7-b7e2-dc14122245ce" Oct 30 10:44:48.044381 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:48.044303 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fk5zl" podUID="e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8" Oct 30 10:44:48.044381 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:48.044372 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vskj5" podUID="23abd71b-1521-4aec-9d79-198f0f8e14a2" Oct 30 10:44:48.274227 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:48.274194 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" event={"ID":"63196a40-0beb-4a8b-b396-0a78f57560f3","Type":"ContainerStarted","Data":"2b106445940cd4e9c2e6f830b14aac4288479bbd4e7b96e16defea30bd8da41d"} Oct 30 10:44:48.274624 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:48.274259 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" event={"ID":"63196a40-0beb-4a8b-b396-0a78f57560f3","Type":"ContainerStarted","Data":"2a580480a0afde71970dd5ce9f51b8d390d759ae506b1f98e32cb28ff46a5275"} Oct 30 10:44:50.044810 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:50.044781 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:50.045251 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:50.044910 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:44:50.045251 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:50.044925 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cgfp" podUID="b06a8e3b-6db3-4cc7-b7e2-dc14122245ce" Oct 30 10:44:50.045251 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:50.044967 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fk5zl" podUID="e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8" Oct 30 10:44:50.045251 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:50.045014 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:50.045251 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:50.045100 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vskj5" podUID="23abd71b-1521-4aec-9d79-198f0f8e14a2" Oct 30 10:44:50.281421 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:50.281389 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" event={"ID":"63196a40-0beb-4a8b-b396-0a78f57560f3","Type":"ContainerStarted","Data":"6753377da187c979ae4179ecfe1c749fe370a897cca784e720f27671937a8d05"} Oct 30 10:44:52.044731 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:52.044691 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:52.044731 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:52.044733 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:52.045400 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:52.044739 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:44:52.045400 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:52.044835 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vskj5" podUID="23abd71b-1521-4aec-9d79-198f0f8e14a2" Oct 30 10:44:52.045400 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:52.044970 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cgfp" podUID="b06a8e3b-6db3-4cc7-b7e2-dc14122245ce" Oct 30 10:44:52.045400 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:52.045033 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fk5zl" podUID="e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8" Oct 30 10:44:52.288043 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:52.288011 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" event={"ID":"63196a40-0beb-4a8b-b396-0a78f57560f3","Type":"ContainerStarted","Data":"5ee81bb626db5bb0ef8aa2e23dae687c9e8e7fa8d69adfbd456d092aa02e11b7"} Oct 30 10:44:52.288313 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:52.288291 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:52.288467 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:52.288346 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:52.302635 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:52.302581 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:52.382263 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:52.382216 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" podStartSLOduration=7.68302819 podStartE2EDuration="22.382203196s" totalImagesPullingTime="14.699175006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 10:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:44:52.368295394 +0000 UTC m=+22.790310695" watchObservedRunningTime="2026-10-30 10:44:52.382203196 +0000 UTC m=+22.804218498" Oct 30 10:44:53.290836 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:53.290704 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:53.306193 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:53.306171 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:44:53.453463 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:53.453398 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:44:53.546784 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:53.546679 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vskj5"] Oct 30 10:44:53.546922 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:53.546805 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:53.546987 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:53.546911 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vskj5" podUID="23abd71b-1521-4aec-9d79-198f0f8e14a2" Oct 30 10:44:53.550252 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:53.550229 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fk5zl"] Oct 30 10:44:53.550357 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:53.550318 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:44:53.550403 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:53.550387 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fk5zl" podUID="e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8" Oct 30 10:44:53.550697 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:53.550682 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4cgfp"] Oct 30 10:44:53.550809 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:53.550777 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:53.550873 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:53.550857 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cgfp" podUID="b06a8e3b-6db3-4cc7-b7e2-dc14122245ce" Oct 30 10:44:53.675268 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:53.675237 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:44:55.044422 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:55.044392 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:44:55.044852 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:55.044538 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:55.044852 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:55.044549 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fk5zl" podUID="e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8" Oct 30 10:44:55.044852 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:55.044566 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:55.044852 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:55.044623 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vskj5" podUID="23abd71b-1521-4aec-9d79-198f0f8e14a2" Oct 30 10:44:55.044852 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:55.044689 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cgfp" podUID="b06a8e3b-6db3-4cc7-b7e2-dc14122245ce" Oct 30 10:44:55.892736 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:55.892668 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:44:56.051209 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:56.051169 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:44:56.291714 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:56.291683 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:44:56.456566 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:56.456535 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:44:57.044216 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:57.044182 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:44:57.044528 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:57.044280 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:57.044528 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:57.044298 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fk5zl" podUID="e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8" Oct 30 10:44:57.044528 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:57.044365 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vskj5" podUID="23abd71b-1521-4aec-9d79-198f0f8e14a2" Oct 30 10:44:57.044528 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:57.044377 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:57.044528 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:57.044480 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cgfp" podUID="b06a8e3b-6db3-4cc7-b7e2-dc14122245ce" Oct 30 10:44:58.642492 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:58.642455 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:44:58.803660 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:58.803629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:44:58.962703 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:58.962631 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:44:59.043721 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:59.043695 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:44:59.043721 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:59.043710 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:44:59.043925 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:59.043704 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:44:59.043925 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:59.043784 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fk5zl" podUID="e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8" Oct 30 10:44:59.043925 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:59.043871 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vskj5" podUID="23abd71b-1521-4aec-9d79-198f0f8e14a2" Oct 30 10:44:59.044031 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:44:59.043942 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4cgfp" podUID="b06a8e3b-6db3-4cc7-b7e2-dc14122245ce" Oct 30 10:44:59.372722 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:59.372692 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:44:59.529878 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:59.529852 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:44:59.738465 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:59.738424 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:44:59.887805 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:59.887772 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeReady" Oct 30 10:44:59.888012 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:59.887892 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 10:44:59.926902 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:44:59.926874 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:00.105832 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:00.105755 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:00.293893 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:00.293865 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:01.044797 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:01.044763 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:45:01.045264 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:01.044886 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:45:01.045264 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:01.044774 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:45:01.062581 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:01.062564 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 10:45:01.062696 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:01.062597 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 10:45:01.062850 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:01.062758 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 10:45:01.062909 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:01.062888 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c9pg5\"" Oct 30 10:45:01.075853 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:01.075831 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-55bss\"" Oct 30 10:45:01.075963 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:01.075911 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 10:45:01.076025 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:01.075989 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vmnnx\"" Oct 30 10:45:01.757597 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:01.757560 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8xmhg" Oct 30 10:45:01.757784 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:01.757689 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 10:45:01.758146 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:01.758127 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8xmhg" Oct 30 10:45:03.049192 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:03.049158 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:03.287128 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:03.287099 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:03.597275 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:03.597246 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:03.787844 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:03.787761 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/23abd71b-1521-4aec-9d79-198f0f8e14a2-original-pull-secret\") pod \"global-pull-secret-syncer-vskj5\" (UID: \"23abd71b-1521-4aec-9d79-198f0f8e14a2\") " pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:45:03.787844 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:03.787808 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs\") pod \"network-metrics-daemon-4cgfp\" (UID: \"b06a8e3b-6db3-4cc7-b7e2-dc14122245ce\") " pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:45:03.788012 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:45:03.787915 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 10:45:03.788012 ip-10-0-137-242 kubenswrapper[2579]: E1030 10:45:03.787984 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs podName:b06a8e3b-6db3-4cc7-b7e2-dc14122245ce nodeName:}" failed. No retries permitted until 2026-10-30 10:45:35.787967045 +0000 UTC m=+66.209982322 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs") pod "network-metrics-daemon-4cgfp" (UID: "b06a8e3b-6db3-4cc7-b7e2-dc14122245ce") : secret "metrics-daemon-secret" not found Oct 30 10:45:03.791152 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:03.791130 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/23abd71b-1521-4aec-9d79-198f0f8e14a2-original-pull-secret\") pod \"global-pull-secret-syncer-vskj5\" (UID: \"23abd71b-1521-4aec-9d79-198f0f8e14a2\") " pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:45:03.799882 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:03.799859 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:03.882651 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:03.882622 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:45:03.888891 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:03.888867 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp4mk\" (UniqueName: \"kubernetes.io/projected/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8-kube-api-access-lp4mk\") pod \"network-check-target-fk5zl\" (UID: \"e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8\") " pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:45:03.891223 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:03.891195 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp4mk\" (UniqueName: \"kubernetes.io/projected/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8-kube-api-access-lp4mk\") pod \"network-check-target-fk5zl\" (UID: \"e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8\") " pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:45:03.965298 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:03.965273 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:04.131008 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:04.130972 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:45:04.312409 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:04.312379 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vskj5"] Oct 30 10:45:04.315624 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:45:04.315595 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23abd71b_1521_4aec_9d79_198f0f8e14a2.slice/crio-08847f14edf9826f66a92064b7ca373f261cb2f09f834c0d68046b9f0f721f9b WatchSource:0}: Error finding container 08847f14edf9826f66a92064b7ca373f261cb2f09f834c0d68046b9f0f721f9b: Status 404 returned error can't find the container with id 08847f14edf9826f66a92064b7ca373f261cb2f09f834c0d68046b9f0f721f9b Oct 30 10:45:04.353083 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:04.353055 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:04.435232 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:04.435202 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fk5zl"] Oct 30 10:45:04.439554 ip-10-0-137-242 kubenswrapper[2579]: W1030 10:45:04.439525 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode66d9bf9_ae24_4a8c_bf21_03a35d3f2ec8.slice/crio-a5268532eb0f982280cbc1c197e09c3907f54bda7245bc79080cd9ac2fbeda6f WatchSource:0}: Error finding container a5268532eb0f982280cbc1c197e09c3907f54bda7245bc79080cd9ac2fbeda6f: Status 404 returned error can't find the container with id a5268532eb0f982280cbc1c197e09c3907f54bda7245bc79080cd9ac2fbeda6f Oct 30 10:45:05.131319 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:05.131279 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:05.315842 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:05.315805 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vskj5" event={"ID":"23abd71b-1521-4aec-9d79-198f0f8e14a2","Type":"ContainerStarted","Data":"08847f14edf9826f66a92064b7ca373f261cb2f09f834c0d68046b9f0f721f9b"} Oct 30 10:45:05.317354 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:05.317329 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fk5zl" event={"ID":"e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8","Type":"ContainerStarted","Data":"a5268532eb0f982280cbc1c197e09c3907f54bda7245bc79080cd9ac2fbeda6f"} Oct 30 10:45:05.531625 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:05.531591 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:05.722739 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:05.722709 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:05.925457 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:05.925355 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:06.322665 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:06.322619 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:06.645736 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:06.645652 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gll67_c64ab741-bc4f-4006-932a-214cf53b6dc3/dns-node-resolver/0.log" Oct 30 10:45:06.719691 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:06.719651 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:06.880485 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:06.880448 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:07.046174 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:07.046119 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:07.219141 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:07.219096 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:07.526273 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:07.526228 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:07.724060 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:07.724031 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:07.846011 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:07.845925 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tvrzn_7437f570-02f5-4131-8b0b-140ed029a668/node-ca/0.log" Oct 30 10:45:08.126025 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:08.125939 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:08.323345 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:08.323312 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:08.326228 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:08.326200 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fk5zl" event={"ID":"e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8","Type":"ContainerStarted","Data":"12ddb4605a1c03ebf2b0fa8f8e5bb0b52ba349c1c2fe6d77e1920d180e4dbc27"} Oct 30 10:45:08.326367 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:08.326350 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:45:08.367828 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:08.367772 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fk5zl" podStartSLOduration=34.944210576 podStartE2EDuration="38.367759578s" totalImagesPullingTime="3.423549002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:45:08.367295153 +0000 UTC m=+38.789310454" watchObservedRunningTime="2026-10-30 10:45:08.367759578 +0000 UTC m=+38.789774878" Oct 30 10:45:08.930550 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:08.930517 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:09.332982 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:09.332944 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vskj5" event={"ID":"23abd71b-1521-4aec-9d79-198f0f8e14a2","Type":"ContainerStarted","Data":"9e642f8c8831497d64ca5bc243e568af9c4705e9f8271d6f395c1c76f14e21ef"} Oct 30 10:45:09.373925 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:09.373862 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vskj5" podStartSLOduration=34.570722809 podStartE2EDuration="39.373845492s" totalImagesPullingTime="4.803122683s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 10:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:45:09.373656625 +0000 UTC m=+39.795671925" watchObservedRunningTime="2026-10-30 10:45:09.373845492 +0000 UTC m=+39.795860794" Oct 30 10:45:09.434204 ip-10-0-137-242 kubenswrapper[2579]: I1030 10:45:09.434041 2579 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 10:45:09.434195 ip-10-0-137-242 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 10:45:09.459809 ip-10-0-137-242 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 10:45:09.459975 ip-10-0-137-242 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 10:45:09.460022 ip-10-0-137-242 systemd[1]: kubelet.service: Consumed 2.088s CPU time. Oct 30 10:45:09.470103 ip-10-0-137-242 systemd[1]: Starting Kubernetes Kubelet... Oct 30 10:45:09.542063 ip-10-0-137-242 kubenswrapper[4289]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 10:45:09.542063 ip-10-0-137-242 kubenswrapper[4289]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 10:45:09.542063 ip-10-0-137-242 kubenswrapper[4289]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 10:45:09.542063 ip-10-0-137-242 kubenswrapper[4289]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 10:45:09.544368 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544282 4289 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 10:45:09.544368 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544362 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 10:45:09.544368 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544368 4289 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 10:45:09.544368 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544372 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 10:45:09.544368 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544376 4289 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 10:45:09.544699 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544381 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 10:45:09.544699 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544385 4289 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 10:45:09.544699 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544389 4289 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 10:45:09.544699 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544393 4289 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 10:45:09.544699 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544397 4289 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 10:45:09.544699 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544401 4289 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 10:45:09.544699 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544404 4289 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 10:45:09.544699 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544408 4289 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 10:45:09.544699 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544412 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 10:45:09.544699 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544416 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 10:45:09.544699 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544447 4289 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 10:45:09.544699 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544452 4289 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 10:45:09.544699 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544456 4289 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 10:45:09.544699 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544460 4289 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 10:45:09.544699 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544465 4289 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 10:45:09.544699 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544469 4289 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 10:45:09.544699 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544472 4289 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 10:45:09.544699 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544476 4289 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 10:45:09.544699 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544481 4289 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 10:45:09.545536 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544485 4289 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 10:45:09.545536 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544489 4289 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 10:45:09.545536 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544493 4289 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 10:45:09.545536 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544497 4289 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 10:45:09.545536 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544501 4289 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 10:45:09.545536 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544505 4289 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 10:45:09.545536 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544510 4289 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 10:45:09.545536 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544514 4289 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 10:45:09.545536 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544517 4289 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 10:45:09.545536 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544521 4289 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 10:45:09.545536 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544526 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 10:45:09.545536 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544531 4289 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 10:45:09.545536 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544535 4289 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 10:45:09.545536 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544543 4289 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 10:45:09.545536 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544553 4289 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 10:45:09.545536 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544558 4289 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 10:45:09.545536 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544563 4289 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 10:45:09.545536 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544567 4289 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 10:45:09.545536 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544572 4289 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 10:45:09.546442 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544577 4289 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 10:45:09.546442 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544583 4289 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 10:45:09.546442 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544587 4289 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 10:45:09.546442 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544592 4289 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 10:45:09.546442 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544596 4289 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 10:45:09.546442 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544601 4289 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 10:45:09.546442 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544605 4289 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 10:45:09.546442 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544609 4289 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 10:45:09.546442 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544613 4289 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 10:45:09.546442 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544617 4289 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 10:45:09.546442 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544623 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 10:45:09.546442 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544627 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 10:45:09.546442 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544633 4289 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 10:45:09.546442 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544637 4289 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 10:45:09.546442 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544642 4289 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 10:45:09.546442 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544646 4289 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 10:45:09.546442 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544651 4289 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 10:45:09.546442 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544655 4289 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 10:45:09.546442 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544660 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 10:45:09.546442 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544664 4289 feature_gate.go:386] unrecognized feature gate: Example Oct 30 10:45:09.547295 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544669 4289 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 10:45:09.547295 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544674 4289 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 10:45:09.547295 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544678 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 10:45:09.547295 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544683 4289 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 10:45:09.547295 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544687 4289 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 10:45:09.547295 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544691 4289 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 10:45:09.547295 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544696 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 10:45:09.547295 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544700 4289 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 10:45:09.547295 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544704 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 10:45:09.547295 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544709 4289 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 10:45:09.547295 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544713 4289 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 10:45:09.547295 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544718 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 10:45:09.547295 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544722 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 10:45:09.547295 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544726 4289 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 10:45:09.547295 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544732 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 10:45:09.547295 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544737 4289 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 10:45:09.547295 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544741 4289 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 10:45:09.547295 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544745 4289 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 10:45:09.547295 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544749 4289 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 10:45:09.547295 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544754 4289 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 10:45:09.548206 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544758 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 10:45:09.548206 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544762 4289 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 10:45:09.548206 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544766 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 10:45:09.548206 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544771 4289 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 10:45:09.548206 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544775 4289 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 10:45:09.548206 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544779 4289 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 10:45:09.548206 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544786 4289 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 10:45:09.548206 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544792 4289 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 10:45:09.548206 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544797 4289 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 10:45:09.548206 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544803 4289 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 10:45:09.548206 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544807 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 10:45:09.548206 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544811 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 10:45:09.548206 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544818 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 10:45:09.548206 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544824 4289 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 10:45:09.548206 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544828 4289 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 10:45:09.548206 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544833 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 10:45:09.548206 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544838 4289 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 10:45:09.548206 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544842 4289 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 10:45:09.548206 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544847 4289 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 10:45:09.549092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544851 4289 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 10:45:09.549092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544855 4289 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 10:45:09.549092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544859 4289 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 10:45:09.549092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.544864 4289 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 10:45:09.549092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546118 4289 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 10:45:09.549092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546132 4289 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 10:45:09.549092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546136 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 10:45:09.549092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546141 4289 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 10:45:09.549092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546146 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 10:45:09.549092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546151 4289 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 10:45:09.549092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546156 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 10:45:09.549092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546160 4289 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 10:45:09.549092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546164 4289 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 10:45:09.549092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546169 4289 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 10:45:09.549092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546173 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 10:45:09.549092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546178 4289 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 10:45:09.549092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546183 4289 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 10:45:09.549092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546187 4289 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 10:45:09.549092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546192 4289 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 10:45:09.549957 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546196 4289 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 10:45:09.549957 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546200 4289 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 10:45:09.549957 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546205 4289 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 10:45:09.549957 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546209 4289 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 10:45:09.549957 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546214 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 10:45:09.549957 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546218 4289 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 10:45:09.549957 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546222 4289 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 10:45:09.549957 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546226 4289 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 10:45:09.549957 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546231 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 10:45:09.549957 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546235 4289 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 10:45:09.549957 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546239 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 10:45:09.549957 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546243 4289 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 10:45:09.549957 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546247 4289 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 10:45:09.549957 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546252 4289 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 10:45:09.549957 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546256 4289 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 10:45:09.549957 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546260 4289 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 10:45:09.549957 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546264 4289 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 10:45:09.549957 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546269 4289 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 10:45:09.549957 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546273 4289 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 10:45:09.549957 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546277 4289 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 10:45:09.550821 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546281 4289 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 10:45:09.550821 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546285 4289 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 10:45:09.550821 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546290 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 10:45:09.550821 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546295 4289 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 10:45:09.550821 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546299 4289 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 10:45:09.550821 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546303 4289 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 10:45:09.550821 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546307 4289 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 10:45:09.550821 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546312 4289 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 10:45:09.550821 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546316 4289 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 10:45:09.550821 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546320 4289 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 10:45:09.550821 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546325 4289 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 10:45:09.550821 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546329 4289 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 10:45:09.550821 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546334 4289 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 10:45:09.550821 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546338 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 10:45:09.550821 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546343 4289 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 10:45:09.550821 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546347 4289 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 10:45:09.550821 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546351 4289 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 10:45:09.550821 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546355 4289 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 10:45:09.550821 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546361 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 10:45:09.551315 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546366 4289 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 10:45:09.551315 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546373 4289 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 10:45:09.551315 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546379 4289 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 10:45:09.551315 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546383 4289 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 10:45:09.551315 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546389 4289 feature_gate.go:386] unrecognized feature gate: Example Oct 30 10:45:09.551315 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546396 4289 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 10:45:09.551315 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546402 4289 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 10:45:09.551315 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546408 4289 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 10:45:09.551315 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546412 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 10:45:09.551315 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546417 4289 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 10:45:09.551315 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546421 4289 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 10:45:09.551315 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546442 4289 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 10:45:09.551315 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546447 4289 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 10:45:09.551315 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546451 4289 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 10:45:09.551315 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546455 4289 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 10:45:09.551315 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546460 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 10:45:09.551315 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546465 4289 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 10:45:09.551315 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546470 4289 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 10:45:09.551315 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546474 4289 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 10:45:09.551315 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546479 4289 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 10:45:09.551851 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546483 4289 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 10:45:09.551851 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546487 4289 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 10:45:09.551851 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546492 4289 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 10:45:09.551851 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546496 4289 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 10:45:09.551851 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546501 4289 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 10:45:09.551851 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546505 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 10:45:09.551851 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546509 4289 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 10:45:09.551851 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546517 4289 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 10:45:09.551851 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546521 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 10:45:09.551851 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546525 4289 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 10:45:09.551851 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546529 4289 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 10:45:09.551851 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546534 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 10:45:09.551851 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546539 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 10:45:09.551851 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546543 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 10:45:09.551851 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546547 4289 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 10:45:09.551851 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546552 4289 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 10:45:09.551851 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546556 4289 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 10:45:09.551851 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546560 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 10:45:09.551851 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546564 4289 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 10:45:09.551851 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546569 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 10:45:09.552348 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546573 4289 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 10:45:09.552348 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546578 4289 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 10:45:09.552348 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546582 4289 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 10:45:09.552348 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546586 4289 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 10:45:09.552348 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546590 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 10:45:09.552348 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546595 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 10:45:09.552348 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546599 4289 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 10:45:09.552348 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546603 4289 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 10:45:09.552348 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546609 4289 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 10:45:09.552348 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546614 4289 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 10:45:09.552348 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546618 4289 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 10:45:09.552348 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.546623 4289 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 10:45:09.552348 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547461 4289 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 10:45:09.552348 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547470 4289 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 10:45:09.552348 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547475 4289 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 10:45:09.552348 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547479 4289 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 10:45:09.552348 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547484 4289 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 10:45:09.552348 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547488 4289 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 10:45:09.552348 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547493 4289 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 10:45:09.552348 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547499 4289 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 10:45:09.552915 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547504 4289 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 10:45:09.552915 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547508 4289 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 10:45:09.552915 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547512 4289 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 10:45:09.552915 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547516 4289 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 10:45:09.552915 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547522 4289 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 10:45:09.552915 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547526 4289 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 10:45:09.552915 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547530 4289 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 10:45:09.552915 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547535 4289 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 10:45:09.552915 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547539 4289 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 10:45:09.552915 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547543 4289 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 10:45:09.552915 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547547 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 10:45:09.552915 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547551 4289 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 10:45:09.552915 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547555 4289 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 10:45:09.552915 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547559 4289 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 10:45:09.552915 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547564 4289 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 10:45:09.552915 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547568 4289 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 10:45:09.552915 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547572 4289 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 10:45:09.552915 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547576 4289 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 10:45:09.552915 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547581 4289 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 10:45:09.552915 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547585 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 10:45:09.553412 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547590 4289 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 10:45:09.553412 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547595 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 10:45:09.553412 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547600 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 10:45:09.553412 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547604 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 10:45:09.553412 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547609 4289 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 10:45:09.553412 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547613 4289 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 10:45:09.553412 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547617 4289 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 10:45:09.553412 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547622 4289 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 10:45:09.553412 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547626 4289 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 10:45:09.553412 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547630 4289 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 10:45:09.553412 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547634 4289 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 10:45:09.553412 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547640 4289 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 10:45:09.553412 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547644 4289 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 10:45:09.553412 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547649 4289 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 10:45:09.553412 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547653 4289 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 10:45:09.553412 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547658 4289 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 10:45:09.553412 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547664 4289 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 10:45:09.553412 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547669 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 10:45:09.553412 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547673 4289 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 10:45:09.554092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547680 4289 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 10:45:09.554092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547687 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 10:45:09.554092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547692 4289 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 10:45:09.554092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547697 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 10:45:09.554092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547702 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 10:45:09.554092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547706 4289 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 10:45:09.554092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547710 4289 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 10:45:09.554092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547714 4289 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 10:45:09.554092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547719 4289 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 10:45:09.554092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547723 4289 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 10:45:09.554092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547727 4289 feature_gate.go:386] unrecognized feature gate: Example Oct 30 10:45:09.554092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547731 4289 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 10:45:09.554092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547735 4289 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 10:45:09.554092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547741 4289 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 10:45:09.554092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547745 4289 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 10:45:09.554092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547750 4289 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 10:45:09.554092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547754 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 10:45:09.554092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547759 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 10:45:09.554092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547763 4289 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 10:45:09.554092 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547767 4289 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 10:45:09.554638 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547771 4289 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 10:45:09.554638 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547782 4289 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 10:45:09.554638 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547786 4289 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 10:45:09.554638 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547791 4289 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 10:45:09.554638 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547797 4289 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 10:45:09.554638 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547801 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 10:45:09.554638 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547805 4289 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 10:45:09.554638 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547810 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 10:45:09.554638 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547814 4289 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 10:45:09.554638 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547823 4289 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 10:45:09.554638 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547827 4289 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 10:45:09.554638 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547832 4289 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 10:45:09.554638 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547836 4289 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 10:45:09.554638 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547840 4289 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 10:45:09.554638 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547844 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 10:45:09.554638 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547849 4289 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 10:45:09.554638 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547854 4289 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 10:45:09.554638 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547858 4289 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 10:45:09.554638 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547862 4289 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 10:45:09.554638 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547866 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 10:45:09.555392 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547871 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 10:45:09.555392 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547875 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 10:45:09.555392 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547880 4289 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 10:45:09.555392 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547884 4289 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 10:45:09.555392 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547888 4289 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 10:45:09.555392 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547894 4289 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 10:45:09.555392 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547901 4289 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 10:45:09.555392 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547906 4289 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 10:45:09.555392 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547913 4289 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 10:45:09.555392 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547917 4289 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 10:45:09.555392 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547923 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 10:45:09.555392 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547928 4289 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 10:45:09.555392 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547932 4289 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 10:45:09.555392 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547936 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 10:45:09.555392 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547940 4289 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 10:45:09.555392 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547945 4289 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 10:45:09.555392 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547950 4289 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 10:45:09.555392 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547955 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 10:45:09.555392 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.547959 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 10:45:09.555980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548097 4289 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 10:45:09.555980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548109 4289 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 10:45:09.555980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548121 4289 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 10:45:09.555980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548128 4289 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 10:45:09.555980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548135 4289 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 10:45:09.555980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548140 4289 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 10:45:09.555980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548146 4289 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 10:45:09.555980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548153 4289 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 10:45:09.555980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548158 4289 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 10:45:09.555980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548163 4289 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 10:45:09.555980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548168 4289 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 10:45:09.555980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548174 4289 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 10:45:09.555980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548179 4289 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 10:45:09.555980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548184 4289 flags.go:64] FLAG: --cgroup-root="" Oct 30 10:45:09.555980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548189 4289 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 10:45:09.555980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548193 4289 flags.go:64] FLAG: --client-ca-file="" Oct 30 10:45:09.555980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548198 4289 flags.go:64] FLAG: --cloud-provider="external" Oct 30 10:45:09.555980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548202 4289 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 10:45:09.555980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548208 4289 flags.go:64] FLAG: --cluster-domain="" Oct 30 10:45:09.555980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548213 4289 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 10:45:09.555980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548218 4289 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 10:45:09.555980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548222 4289 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 10:45:09.555980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548228 4289 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 10:45:09.555980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548235 4289 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548240 4289 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548246 4289 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548251 4289 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548255 4289 flags.go:64] FLAG: --contention-profiling="false" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548260 4289 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548265 4289 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548270 4289 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548275 4289 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548282 4289 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548287 4289 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548294 4289 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548299 4289 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548304 4289 flags.go:64] FLAG: --enable-server="true" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548309 4289 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548317 4289 flags.go:64] FLAG: --event-burst="100" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548322 4289 flags.go:64] FLAG: --event-qps="50" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548327 4289 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548332 4289 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548336 4289 flags.go:64] FLAG: --eviction-hard="" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548343 4289 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548348 4289 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548352 4289 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548357 4289 flags.go:64] FLAG: --eviction-soft="" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548362 4289 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 10:45:09.556604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548367 4289 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 10:45:09.557213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548372 4289 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 10:45:09.557213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548377 4289 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 10:45:09.557213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548381 4289 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 10:45:09.557213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548386 4289 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 10:45:09.557213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548390 4289 flags.go:64] FLAG: --feature-gates="" Oct 30 10:45:09.557213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548396 4289 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 10:45:09.557213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548412 4289 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 10:45:09.557213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548418 4289 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 10:45:09.557213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548440 4289 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 10:45:09.557213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548446 4289 flags.go:64] FLAG: --healthz-port="10248" Oct 30 10:45:09.557213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548451 4289 flags.go:64] FLAG: --help="false" Oct 30 10:45:09.557213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548456 4289 flags.go:64] FLAG: --hostname-override="ip-10-0-137-242.ec2.internal" Oct 30 10:45:09.557213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548461 4289 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 10:45:09.557213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548466 4289 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 10:45:09.557213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548471 4289 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 10:45:09.557213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548476 4289 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 10:45:09.557213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548482 4289 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 10:45:09.557213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548489 4289 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 10:45:09.557213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548494 4289 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 10:45:09.557213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548498 4289 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 10:45:09.557213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548503 4289 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 10:45:09.557213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548508 4289 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 10:45:09.557213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548513 4289 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 10:45:09.557213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548517 4289 flags.go:64] FLAG: --kube-reserved="" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548522 4289 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548527 4289 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548533 4289 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548537 4289 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548542 4289 flags.go:64] FLAG: --lock-file="" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548546 4289 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548551 4289 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548556 4289 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548565 4289 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548570 4289 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548575 4289 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548580 4289 flags.go:64] FLAG: --logging-format="text" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548585 4289 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548590 4289 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548595 4289 flags.go:64] FLAG: --manifest-url="" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548601 4289 flags.go:64] FLAG: --manifest-url-header="" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548607 4289 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548612 4289 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548619 4289 flags.go:64] FLAG: --max-pods="110" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548624 4289 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548629 4289 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548634 4289 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548638 4289 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548644 4289 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 10:45:09.557855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548648 4289 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548654 4289 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548666 4289 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548671 4289 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548676 4289 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548681 4289 flags.go:64] FLAG: --pod-cidr="" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548685 4289 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548689 4289 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548695 4289 flags.go:64] FLAG: --pods-per-core="0" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548699 4289 flags.go:64] FLAG: --port="10250" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548705 4289 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548709 4289 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d0210ea6e5c5c0c5" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548715 4289 flags.go:64] FLAG: --qos-reserved="" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548719 4289 flags.go:64] FLAG: --read-only-port="10255" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548725 4289 flags.go:64] FLAG: --register-node="true" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548729 4289 flags.go:64] FLAG: --register-with-taints="" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548735 4289 flags.go:64] FLAG: --registry-burst="10" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548740 4289 flags.go:64] FLAG: --registry-qps="5" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548744 4289 flags.go:64] FLAG: --reserved-cpus="" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548749 4289 flags.go:64] FLAG: --reserved-memory="" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548762 4289 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548767 4289 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548772 4289 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548776 4289 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548789 4289 flags.go:64] FLAG: --runonce="false" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548794 4289 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 10:45:09.558473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548800 4289 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548805 4289 flags.go:64] FLAG: --seccomp-default="false" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548809 4289 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548814 4289 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548819 4289 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548824 4289 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548829 4289 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548835 4289 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548840 4289 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548845 4289 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548850 4289 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548855 4289 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548860 4289 flags.go:64] FLAG: --system-cgroups="" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548864 4289 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548872 4289 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548878 4289 flags.go:64] FLAG: --tls-cert-file="" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548882 4289 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548893 4289 flags.go:64] FLAG: --tls-min-version="" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548898 4289 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548902 4289 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548907 4289 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548911 4289 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548916 4289 flags.go:64] FLAG: --v="2" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548922 4289 flags.go:64] FLAG: --version="false" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548929 4289 flags.go:64] FLAG: --vmodule="" Oct 30 10:45:09.559086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548935 4289 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 10:45:09.559745 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.548941 4289 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 10:45:09.559745 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549165 4289 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 10:45:09.559745 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549174 4289 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 10:45:09.559745 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549180 4289 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 10:45:09.559745 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549185 4289 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 10:45:09.559745 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549194 4289 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 10:45:09.559745 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549198 4289 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 10:45:09.559745 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549202 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 10:45:09.559745 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549207 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 10:45:09.559745 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549211 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 10:45:09.559745 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549216 4289 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 10:45:09.559745 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549221 4289 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 10:45:09.559745 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549225 4289 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 10:45:09.559745 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549229 4289 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 10:45:09.559745 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549234 4289 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 10:45:09.559745 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549238 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 10:45:09.559745 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549242 4289 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 10:45:09.559745 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549247 4289 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 10:45:09.559745 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549251 4289 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 10:45:09.559745 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549256 4289 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 10:45:09.559745 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549260 4289 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 10:45:09.560306 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549265 4289 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 10:45:09.560306 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549269 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 10:45:09.560306 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549273 4289 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 10:45:09.560306 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549278 4289 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 10:45:09.560306 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549282 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 10:45:09.560306 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549286 4289 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 10:45:09.560306 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549290 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 10:45:09.560306 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549294 4289 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 10:45:09.560306 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549299 4289 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 10:45:09.560306 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549303 4289 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 10:45:09.560306 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549307 4289 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 10:45:09.560306 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549312 4289 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 10:45:09.560306 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549316 4289 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 10:45:09.560306 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549320 4289 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 10:45:09.560306 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549324 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 10:45:09.560306 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549328 4289 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 10:45:09.560306 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549335 4289 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 10:45:09.560306 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549348 4289 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 10:45:09.560306 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549353 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 10:45:09.560894 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549357 4289 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 10:45:09.560894 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549362 4289 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 10:45:09.560894 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549366 4289 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 10:45:09.560894 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549370 4289 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 10:45:09.560894 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549375 4289 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 10:45:09.560894 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549379 4289 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 10:45:09.560894 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549383 4289 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 10:45:09.560894 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549388 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 10:45:09.560894 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549392 4289 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 10:45:09.560894 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549396 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 10:45:09.560894 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549401 4289 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 10:45:09.560894 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549405 4289 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 10:45:09.560894 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549409 4289 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 10:45:09.560894 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549413 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 10:45:09.560894 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549418 4289 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 10:45:09.560894 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549422 4289 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 10:45:09.560894 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549443 4289 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 10:45:09.560894 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549448 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 10:45:09.560894 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549453 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 10:45:09.561371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549457 4289 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 10:45:09.561371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549461 4289 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 10:45:09.561371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549465 4289 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 10:45:09.561371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549470 4289 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 10:45:09.561371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549474 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 10:45:09.561371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549479 4289 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 10:45:09.561371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549483 4289 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 10:45:09.561371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549487 4289 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 10:45:09.561371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549491 4289 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 10:45:09.561371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549496 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 10:45:09.561371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549502 4289 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 10:45:09.561371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549506 4289 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 10:45:09.561371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549518 4289 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 10:45:09.561371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549523 4289 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 10:45:09.561371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549527 4289 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 10:45:09.561371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549532 4289 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 10:45:09.561371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549536 4289 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 10:45:09.561371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549540 4289 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 10:45:09.561371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549545 4289 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 10:45:09.561371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549549 4289 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 10:45:09.561891 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549554 4289 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 10:45:09.561891 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549570 4289 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 10:45:09.561891 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549578 4289 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 10:45:09.561891 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549583 4289 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 10:45:09.561891 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549588 4289 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 10:45:09.561891 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549592 4289 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 10:45:09.561891 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549599 4289 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 10:45:09.561891 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549605 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 10:45:09.561891 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549610 4289 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 10:45:09.561891 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549615 4289 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 10:45:09.561891 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549620 4289 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 10:45:09.561891 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549624 4289 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 10:45:09.561891 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549628 4289 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 10:45:09.561891 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549633 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 10:45:09.561891 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549637 4289 feature_gate.go:386] unrecognized feature gate: Example Oct 30 10:45:09.561891 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549641 4289 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 10:45:09.561891 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549645 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 10:45:09.561891 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549650 4289 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 10:45:09.561891 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549654 4289 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 10:45:09.562391 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549658 4289 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 10:45:09.562391 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549663 4289 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 10:45:09.562391 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549667 4289 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 10:45:09.562391 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549674 4289 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 10:45:09.562391 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549678 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 10:45:09.562391 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549682 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 10:45:09.562391 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549694 4289 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 10:45:09.562391 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549699 4289 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 10:45:09.562391 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549703 4289 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 10:45:09.562391 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.549746 4289 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 10:45:09.562391 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549803 4289 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 10:45:09.562391 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549810 4289 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 10:45:09.562391 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549815 4289 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 10:45:09.562391 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549820 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 10:45:09.562391 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549825 4289 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 10:45:09.562391 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549829 4289 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 10:45:09.563003 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549834 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 10:45:09.563003 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549838 4289 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 10:45:09.563003 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549842 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 10:45:09.563003 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549846 4289 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 10:45:09.563003 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549850 4289 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 10:45:09.563003 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549855 4289 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 10:45:09.563003 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549859 4289 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 10:45:09.563003 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549863 4289 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 10:45:09.563003 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549867 4289 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 10:45:09.563003 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549871 4289 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 10:45:09.563003 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549875 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 10:45:09.563003 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549881 4289 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 10:45:09.563003 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549886 4289 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 10:45:09.563003 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549890 4289 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 10:45:09.563003 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549894 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 10:45:09.563003 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549899 4289 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 10:45:09.563003 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549903 4289 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 10:45:09.563003 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549907 4289 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 10:45:09.563003 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549911 4289 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 10:45:09.563525 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549916 4289 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 10:45:09.563525 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549922 4289 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 10:45:09.563525 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549927 4289 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 10:45:09.563525 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549931 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 10:45:09.563525 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549943 4289 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 10:45:09.563525 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549948 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 10:45:09.563525 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549952 4289 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 10:45:09.563525 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549956 4289 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 10:45:09.563525 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549960 4289 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 10:45:09.563525 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549964 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 10:45:09.563525 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549969 4289 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 10:45:09.563525 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549973 4289 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 10:45:09.563525 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549977 4289 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 10:45:09.563525 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549981 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 10:45:09.563525 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549986 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 10:45:09.563525 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549990 4289 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 10:45:09.563525 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549994 4289 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 10:45:09.563525 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.549999 4289 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 10:45:09.563525 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550003 4289 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 10:45:09.564055 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550007 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 10:45:09.564055 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550011 4289 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 10:45:09.564055 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550015 4289 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 10:45:09.564055 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550019 4289 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 10:45:09.564055 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550024 4289 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 10:45:09.564055 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550030 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 10:45:09.564055 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550034 4289 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 10:45:09.564055 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550038 4289 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 10:45:09.564055 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550043 4289 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 10:45:09.564055 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550047 4289 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 10:45:09.564055 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550052 4289 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 10:45:09.564055 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550056 4289 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 10:45:09.564055 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550060 4289 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 10:45:09.564055 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550064 4289 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 10:45:09.564055 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550070 4289 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 10:45:09.564055 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550075 4289 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 10:45:09.564055 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550080 4289 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 10:45:09.564055 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550088 4289 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 10:45:09.564055 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550104 4289 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 10:45:09.564055 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550109 4289 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 10:45:09.564548 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550114 4289 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 10:45:09.564548 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550119 4289 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 10:45:09.564548 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550125 4289 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 10:45:09.564548 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550131 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 10:45:09.564548 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550136 4289 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 10:45:09.564548 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550140 4289 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 10:45:09.564548 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550145 4289 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 10:45:09.564548 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550149 4289 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 10:45:09.564548 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550153 4289 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 10:45:09.564548 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550157 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 10:45:09.564548 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550162 4289 feature_gate.go:386] unrecognized feature gate: Example Oct 30 10:45:09.564548 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550166 4289 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 10:45:09.564548 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550170 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 10:45:09.564548 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550174 4289 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 10:45:09.564548 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550179 4289 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 10:45:09.564548 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550183 4289 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 10:45:09.564548 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550187 4289 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 10:45:09.564548 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550193 4289 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 10:45:09.564548 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550198 4289 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 10:45:09.565012 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550202 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 10:45:09.565012 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550207 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 10:45:09.565012 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550211 4289 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 10:45:09.565012 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550215 4289 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 10:45:09.565012 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550219 4289 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 10:45:09.565012 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550223 4289 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 10:45:09.565012 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550227 4289 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 10:45:09.565012 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550234 4289 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 10:45:09.565012 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550238 4289 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 10:45:09.565012 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550243 4289 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 10:45:09.565012 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550247 4289 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 10:45:09.565012 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550251 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 10:45:09.565012 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550264 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 10:45:09.565012 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550268 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 10:45:09.565012 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550274 4289 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 10:45:09.565012 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550278 4289 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 10:45:09.565012 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550282 4289 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 10:45:09.565012 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550286 4289 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 10:45:09.565012 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550291 4289 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 10:45:09.565012 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550295 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 10:45:09.565779 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550299 4289 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 10:45:09.565779 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550303 4289 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 10:45:09.565779 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.550308 4289 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 10:45:09.565779 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.554029 4289 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 10:45:09.565779 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.554046 4289 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 10:45:09.565779 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.554064 4289 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 10:45:09.565779 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.554069 4289 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 10:45:09.565779 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554122 4289 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 10:45:09.565779 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554127 4289 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 10:45:09.565779 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554131 4289 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 10:45:09.565779 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554134 4289 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 10:45:09.565779 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554138 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 10:45:09.565779 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554141 4289 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 10:45:09.565779 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554144 4289 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 10:45:09.565779 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554147 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 10:45:09.565779 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554150 4289 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 10:45:09.565779 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554153 4289 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 10:45:09.565779 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554155 4289 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 10:45:09.565779 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554159 4289 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 10:45:09.565779 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554161 4289 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 10:45:09.566416 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554164 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 10:45:09.566416 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554168 4289 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 10:45:09.566416 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554170 4289 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 10:45:09.566416 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554173 4289 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 10:45:09.566416 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554176 4289 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 10:45:09.566416 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554178 4289 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 10:45:09.566416 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554181 4289 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 10:45:09.566416 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554184 4289 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 10:45:09.566416 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554186 4289 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 10:45:09.566416 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554189 4289 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 10:45:09.566416 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554192 4289 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 10:45:09.566416 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554194 4289 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 10:45:09.566416 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554197 4289 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 10:45:09.566416 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554200 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 10:45:09.566416 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554202 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 10:45:09.566416 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554205 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 10:45:09.566416 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554211 4289 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 10:45:09.566416 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554214 4289 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 10:45:09.566416 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554217 4289 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 10:45:09.566416 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554219 4289 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 10:45:09.566920 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554222 4289 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 10:45:09.566920 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554224 4289 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 10:45:09.566920 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554227 4289 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 10:45:09.566920 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554229 4289 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 10:45:09.566920 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554232 4289 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 10:45:09.566920 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554235 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 10:45:09.566920 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554238 4289 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 10:45:09.566920 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554241 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 10:45:09.566920 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554243 4289 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 10:45:09.566920 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554246 4289 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 10:45:09.566920 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554249 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 10:45:09.566920 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554251 4289 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 10:45:09.566920 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554254 4289 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 10:45:09.566920 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554270 4289 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 10:45:09.566920 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554275 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 10:45:09.566920 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554279 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 10:45:09.566920 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554282 4289 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 10:45:09.566920 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554285 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 10:45:09.566920 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554287 4289 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 10:45:09.567371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554290 4289 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 10:45:09.567371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554293 4289 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 10:45:09.567371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554295 4289 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 10:45:09.567371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554298 4289 feature_gate.go:386] unrecognized feature gate: Example Oct 30 10:45:09.567371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554300 4289 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 10:45:09.567371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554303 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 10:45:09.567371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554306 4289 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 10:45:09.567371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554308 4289 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 10:45:09.567371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554311 4289 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 10:45:09.567371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554313 4289 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 10:45:09.567371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554316 4289 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 10:45:09.567371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554319 4289 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 10:45:09.567371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554322 4289 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 10:45:09.567371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554325 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 10:45:09.567371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554327 4289 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 10:45:09.567371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554330 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 10:45:09.567371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554332 4289 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 10:45:09.567371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554335 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 10:45:09.567371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554338 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 10:45:09.567371 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554340 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 10:45:09.567879 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554343 4289 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 10:45:09.567879 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554346 4289 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 10:45:09.567879 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554348 4289 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 10:45:09.567879 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554351 4289 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 10:45:09.567879 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554354 4289 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 10:45:09.567879 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554356 4289 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 10:45:09.567879 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554359 4289 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 10:45:09.567879 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554362 4289 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 10:45:09.567879 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554364 4289 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 10:45:09.567879 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554367 4289 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 10:45:09.567879 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554371 4289 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 10:45:09.567879 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554375 4289 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 10:45:09.567879 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554378 4289 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 10:45:09.567879 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554381 4289 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 10:45:09.567879 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554383 4289 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 10:45:09.567879 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554386 4289 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 10:45:09.567879 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554389 4289 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 10:45:09.567879 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554391 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 10:45:09.567879 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554394 4289 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 10:45:09.567879 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554397 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 10:45:09.568364 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554399 4289 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 10:45:09.568364 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554402 4289 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 10:45:09.568364 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554404 4289 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 10:45:09.568364 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554407 4289 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 10:45:09.568364 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554410 4289 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 10:45:09.568364 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554414 4289 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 10:45:09.568364 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554418 4289 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 10:45:09.568364 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554420 4289 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 10:45:09.568364 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554423 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 10:45:09.568364 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554440 4289 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 10:45:09.568364 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554444 4289 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 10:45:09.568364 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554447 4289 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 10:45:09.568364 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554449 4289 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 10:45:09.568364 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.554452 4289 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 10:45:09.568364 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.554480 4289 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 10:45:09.568735 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.555954 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 10:45:09.568735 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.555970 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 10:45:09.568735 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.555975 4289 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 10:45:09.568735 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.555978 4289 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 10:45:09.568735 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.555982 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 10:45:09.568735 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.555985 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 10:45:09.568735 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.555987 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 10:45:09.568735 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.555990 4289 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 10:45:09.568735 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.555993 4289 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 10:45:09.568735 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.555997 4289 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 10:45:09.568735 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556000 4289 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 10:45:09.568735 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556002 4289 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 10:45:09.568735 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556005 4289 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 10:45:09.568735 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556008 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 10:45:09.568735 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556011 4289 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 10:45:09.568735 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556014 4289 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 10:45:09.568735 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556016 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 10:45:09.568735 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556019 4289 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 10:45:09.568735 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556022 4289 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 10:45:09.569188 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556024 4289 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 10:45:09.569188 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556027 4289 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 10:45:09.569188 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556030 4289 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 10:45:09.569188 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556032 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 10:45:09.569188 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556035 4289 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 10:45:09.569188 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556037 4289 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 10:45:09.569188 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556040 4289 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 10:45:09.569188 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556043 4289 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 10:45:09.569188 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556045 4289 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 10:45:09.569188 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556048 4289 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 10:45:09.569188 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556050 4289 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 10:45:09.569188 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556056 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 10:45:09.569188 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556059 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 10:45:09.569188 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556062 4289 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 10:45:09.569188 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556064 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 10:45:09.569188 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556067 4289 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 10:45:09.569188 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556070 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 10:45:09.569188 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556073 4289 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 10:45:09.569188 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556075 4289 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 10:45:09.569188 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556078 4289 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 10:45:09.569686 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556081 4289 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 10:45:09.569686 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556083 4289 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 10:45:09.569686 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556086 4289 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 10:45:09.569686 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556088 4289 feature_gate.go:386] unrecognized feature gate: Example Oct 30 10:45:09.569686 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556091 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 10:45:09.569686 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556093 4289 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 10:45:09.569686 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556096 4289 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 10:45:09.569686 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556099 4289 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 10:45:09.569686 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556101 4289 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 10:45:09.569686 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556104 4289 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 10:45:09.569686 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556106 4289 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 10:45:09.569686 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556110 4289 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 10:45:09.569686 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556113 4289 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 10:45:09.569686 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556116 4289 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 10:45:09.569686 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556118 4289 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 10:45:09.569686 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556121 4289 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 10:45:09.569686 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556123 4289 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 10:45:09.569686 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556126 4289 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 10:45:09.569686 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556128 4289 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 10:45:09.569686 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556131 4289 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 10:45:09.570170 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556133 4289 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 10:45:09.570170 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556136 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 10:45:09.570170 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556139 4289 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 10:45:09.570170 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556142 4289 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 10:45:09.570170 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556144 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 10:45:09.570170 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556147 4289 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 10:45:09.570170 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556150 4289 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 10:45:09.570170 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556152 4289 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 10:45:09.570170 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556155 4289 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 10:45:09.570170 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556159 4289 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 10:45:09.570170 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556164 4289 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 10:45:09.570170 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556167 4289 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 10:45:09.570170 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556170 4289 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 10:45:09.570170 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556173 4289 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 10:45:09.570170 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556176 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 10:45:09.570170 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556179 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 10:45:09.570170 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556182 4289 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 10:45:09.570170 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556185 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 10:45:09.570170 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556188 4289 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 10:45:09.570673 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556190 4289 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 10:45:09.570673 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556193 4289 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 10:45:09.570673 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556195 4289 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 10:45:09.570673 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556198 4289 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 10:45:09.570673 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556201 4289 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 10:45:09.570673 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556204 4289 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 10:45:09.570673 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556207 4289 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 10:45:09.570673 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556210 4289 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 10:45:09.570673 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556212 4289 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 10:45:09.570673 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556214 4289 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 10:45:09.570673 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556217 4289 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 10:45:09.570673 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556219 4289 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 10:45:09.570673 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556222 4289 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 10:45:09.570673 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556225 4289 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 10:45:09.570673 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556227 4289 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 10:45:09.570673 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556230 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 10:45:09.570673 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556233 4289 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 10:45:09.570673 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556235 4289 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 10:45:09.570673 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556238 4289 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 10:45:09.571322 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556241 4289 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 10:45:09.571322 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556244 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 10:45:09.571322 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556246 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 10:45:09.571322 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556249 4289 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 10:45:09.571322 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556252 4289 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 10:45:09.571322 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556254 4289 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 10:45:09.571322 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556257 4289 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 10:45:09.571322 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556260 4289 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 10:45:09.571322 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556264 4289 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 10:45:09.571322 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.556294 4289 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 10:45:09.571322 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556340 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 10:45:09.571322 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556344 4289 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 10:45:09.571322 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556347 4289 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 10:45:09.571322 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556350 4289 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 10:45:09.571322 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556352 4289 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 10:45:09.571990 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556355 4289 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 10:45:09.571990 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556358 4289 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 10:45:09.571990 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556360 4289 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 10:45:09.571990 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556363 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 10:45:09.571990 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556366 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 10:45:09.571990 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556369 4289 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 10:45:09.571990 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556372 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 10:45:09.571990 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556375 4289 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 10:45:09.571990 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556377 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 10:45:09.571990 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556380 4289 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 10:45:09.571990 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556382 4289 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 10:45:09.571990 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556385 4289 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 10:45:09.571990 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556388 4289 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 10:45:09.571990 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556390 4289 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 10:45:09.571990 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556393 4289 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 10:45:09.571990 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556396 4289 feature_gate.go:386] unrecognized feature gate: Example Oct 30 10:45:09.571990 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556398 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 10:45:09.571990 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556401 4289 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 10:45:09.571990 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556403 4289 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 10:45:09.571990 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556406 4289 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 10:45:09.572761 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556409 4289 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 10:45:09.572761 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556411 4289 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 10:45:09.572761 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556414 4289 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 10:45:09.572761 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556416 4289 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 10:45:09.572761 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556419 4289 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 10:45:09.572761 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556421 4289 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 10:45:09.572761 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556424 4289 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 10:45:09.572761 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556448 4289 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 10:45:09.572761 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556452 4289 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 10:45:09.572761 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556456 4289 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 10:45:09.572761 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556458 4289 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 10:45:09.572761 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556461 4289 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 10:45:09.572761 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556463 4289 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 10:45:09.572761 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556466 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 10:45:09.572761 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556469 4289 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 10:45:09.572761 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556471 4289 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 10:45:09.572761 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556474 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 10:45:09.572761 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556480 4289 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 10:45:09.572761 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556483 4289 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 10:45:09.572761 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556486 4289 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 10:45:09.573342 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556488 4289 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 10:45:09.573342 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556492 4289 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 10:45:09.573342 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556496 4289 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 10:45:09.573342 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556499 4289 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 10:45:09.573342 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556502 4289 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 10:45:09.573342 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556505 4289 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 10:45:09.573342 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556508 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 10:45:09.573342 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556510 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 10:45:09.573342 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556513 4289 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 10:45:09.573342 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556515 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 10:45:09.573342 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556518 4289 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 10:45:09.573342 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556520 4289 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 10:45:09.573342 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556523 4289 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 10:45:09.573342 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556525 4289 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 10:45:09.573342 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556528 4289 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 10:45:09.573342 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556531 4289 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 10:45:09.573342 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556534 4289 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 10:45:09.573342 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556536 4289 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 10:45:09.573342 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556539 4289 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 10:45:09.574061 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556543 4289 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 10:45:09.574061 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556546 4289 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 10:45:09.574061 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556548 4289 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 10:45:09.574061 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556551 4289 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 10:45:09.574061 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556553 4289 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 10:45:09.574061 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556556 4289 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 10:45:09.574061 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556558 4289 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 10:45:09.574061 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556561 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 10:45:09.574061 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556563 4289 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 10:45:09.574061 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556566 4289 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 10:45:09.574061 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556570 4289 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 10:45:09.574061 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556574 4289 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 10:45:09.574061 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556577 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 10:45:09.574061 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556579 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 10:45:09.574061 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556582 4289 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 10:45:09.574061 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556585 4289 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 10:45:09.574061 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556587 4289 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 10:45:09.574061 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556590 4289 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 10:45:09.574061 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556592 4289 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 10:45:09.575106 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556596 4289 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 10:45:09.575106 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556600 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 10:45:09.575106 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556603 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 10:45:09.575106 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556606 4289 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 10:45:09.575106 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556609 4289 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 10:45:09.575106 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556612 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 10:45:09.575106 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556614 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 10:45:09.575106 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556617 4289 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 10:45:09.575106 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556620 4289 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 10:45:09.575106 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556623 4289 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 10:45:09.575106 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556626 4289 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 10:45:09.575106 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556628 4289 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 10:45:09.575106 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556631 4289 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 10:45:09.575106 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556634 4289 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 10:45:09.575106 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556637 4289 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 10:45:09.575106 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556640 4289 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 10:45:09.575106 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556642 4289 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 10:45:09.575106 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556645 4289 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 10:45:09.575106 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556648 4289 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 10:45:09.575997 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556651 4289 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 10:45:09.575997 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556654 4289 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 10:45:09.575997 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556657 4289 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 10:45:09.575997 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:09.556659 4289 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 10:45:09.575997 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.556717 4289 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 10:45:09.575997 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.557685 4289 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 10:45:09.575997 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.557740 4289 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 10:45:09.575997 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.557945 4289 server.go:1014] "Starting client certificate rotation" Oct 30 10:45:09.575997 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.558087 4289 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 10:45:09.575997 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.558133 4289 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 10:39:30 +0000 UTC" deadline="2028-08-15 21:41:35.213745336 +0000 UTC" Oct 30 10:45:09.575997 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.558169 4289 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15730h56m25.655579776s" Oct 30 10:45:09.575997 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.558462 4289 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 10:45:09.575997 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.559523 4289 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 10:45:09.575997 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.560865 4289 log.go:25] "Validated CRI v1 runtime API" Oct 30 10:45:09.575997 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.562314 4289 log.go:25] "Validated CRI v1 image API" Oct 30 10:45:09.575997 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.563111 4289 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 10:45:09.576744 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.564469 4289 fs.go:135] Filesystem UUIDs: map[65f273d2-6938-4a11-89d1-a50ef37222b7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9d148982-660c-4fa9-88e7-bc2b9bae4cb1:/dev/nvme0n1p4] Oct 30 10:45:09.576817 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.564491 4289 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00c3d5844216d347f4d1b516b3a43e4b86b7fa9b97a7a2b78bf8b6ca66c0945a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00c3d5844216d347f4d1b516b3a43e4b86b7fa9b97a7a2b78bf8b6ca66c0945a/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/08847f14edf9826f66a92064b7ca373f261cb2f09f834c0d68046b9f0f721f9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/08847f14edf9826f66a92064b7ca373f261cb2f09f834c0d68046b9f0f721f9b/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d54cc91ee15ed4118ea45d0d6a1cb02b727905f2203c2637af7dc57120f9a67/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d54cc91ee15ed4118ea45d0d6a1cb02b727905f2203c2637af7dc57120f9a67/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1389d1ff3c310f1cf00b31a6d1be34fbf370e4b033f5285d965c957418f67274/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1389d1ff3c310f1cf00b31a6d1be34fbf370e4b033f5285d965c957418f67274/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/149f994b0fde7bb8a2e011e60cf13a1aa60718df8878ae2891567ce7dae652c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/149f994b0fde7bb8a2e011e60cf13a1aa60718df8878ae2891567ce7dae652c6/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f0a9655b588d3a8d23f1e9090caf5b0a8c49ac951b41013762ce64399de158a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f0a9655b588d3a8d23f1e9090caf5b0a8c49ac951b41013762ce64399de158a/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48a8eb8a2666149b54b9c3136708009c599aba6caabd13677a5dfad1e0311aac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48a8eb8a2666149b54b9c3136708009c599aba6caabd13677a5dfad1e0311aac/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c5a6238e5c42ce712e1c576150adf2b35ce71c93d62fddc1a72c8a6a7b515ee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c5a6238e5c42ce712e1c576150adf2b35ce71c93d62fddc1a72c8a6a7b515ee/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/545d644eb3c1bb16fdd8358db707739b2bb7321f97b3f5e75edadc84ba11168f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/545d644eb3c1bb16fdd8358db707739b2bb7321f97b3f5e75edadc84ba11168f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5268532eb0f982280cbc1c197e09c3907f54bda7245bc79080cd9ac2fbeda6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5268532eb0f982280cbc1c197e09c3907f54bda7245bc79080cd9ac2fbeda6f/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2d110367251c32e839f6d0a99742af4fec95018d18f554439b4b7a9f54a920b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2d110367251c32e839f6d0a99742af4fec95018d18f554439b4b7a9f54a920b/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc410460dda0dc3b4b216b489316da2ebc81a3aee74e7461260ed8ae49db6e94/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc410460dda0dc3b4b216b489316da2ebc81a3aee74e7461260ed8ae49db6e94/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23abd71b-1521-4aec-9d79-198f0f8e14a2/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/23abd71b-1521-4aec-9d79-198f0f8e14a2/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/275105df-bf86-436c-a464-890b4510c09d/volumes/kubernetes.io~projected/kube-api-access-rzt9k:{mountpoint:/var/lib/kubelet/pods/275105df-bf86-436c-a464-890b4510c09d/volumes/kubernetes.io~projected/kube-api-access-rzt9k major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45d9d286-4a6a-4ebb-91ab-324335c4a2ab/volumes/kubernetes.io~projected/kube-api-access-q8pl4:{mountpoint:/var/lib/kubelet/pods/45d9d286-4a6a-4ebb-91ab-324335c4a2ab/volumes/kubernetes.io~projected/kube-api-access-q8pl4 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/538de0b2-f736-4723-92d6-e600b9de4a26/volumes/kubernetes.io~projected/kube-api-access-xrb2t:{mountpoint:/var/lib/kubelet/pods/538de0b2-f736-4723-92d6-e600b9de4a26/volumes/kubernetes.io~projected/kube-api-access-xrb2t major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63196a40-0beb-4a8b-b396-0a78f57560f3/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/63196a40-0beb-4a8b-b396-0a78f57560f3/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63196a40-0beb-4a8b-b396-0a78f57560f3/volumes/kubernetes.io~projected/kube-api-access-58d4h:{mountpoint:/var/lib/kubelet/pods/63196a40-0beb-4a8b-b396-0a78f57560f3/volumes/kubernetes.io~projected/kube-api-access-58d4h major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63196a40-0beb-4a8b-b396-0a78f57560f3/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/63196a40-0beb-4a8b-b396-0a78f57560f3/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7437f570-02f5-4131-8b0b-140ed029a668/volumes/kubernetes.io~projected/kube-api-access-pv7hl:{mountpoint:/var/lib/kubelet/pods/7437f570-02f5-4131-8b0b-140ed029a668/volumes/kubernetes.io~projected/kube-api-access-pv7hl major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f26208e-e65a-483d-ba73-d45fe31084bc/volumes/kubernetes.io~projected/kube-api-access-4pzwb:{mountpoint:/var/lib/kubelet/pods/9f26208e-e65a-483d-ba73-d45fe31084bc/volumes/kubernetes.io~projected/kube-api-access-4pzwb major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce/volumes/kubernetes.io~projected/kube-api-access-wqjpq:{mountpoint:/var/lib/kubelet/pods/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce/volumes/kubernetes.io~projected/kube-api-access-wqjpq major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4b288c6-5317-410f-9f88-0352f4164d20/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/c4b288c6-5317-410f-9f88-0352f4164d20/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4b288c6-5317-410f-9f88-0352f4164d20/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/c4b288c6-5317-410f-9f88-0352f4164d20/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4b288c6-5317-410f-9f88-0352f4164d20/volumes/kubernetes.io~projected/kube-api-access-k9g8f:{mountpoint:/var/lib/kubelet/pods/c4b288c6-5317-410f-9f88-0352f4164d20/volumes/kubernetes.io~projected/kube-api-access-k9g8f major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c64ab741-bc4f-4006-932a-214cf53b6dc3/volumes/kubernetes.io~projected/kube-api-access-bflhl:{mountpoint:/var/lib/kubelet/pods/c64ab741-bc4f-4006-932a-214cf53b6dc3/volumes/kubernetes.io~projected/kube-api-access-bflhl major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8/volumes/kubernetes.io~projected/kube-api-access-lp4mk:{mountpoint:/var/lib/kubelet/pods/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8/volumes/kubernetes.io~projected/kube-api-access-lp4mk major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/4f88267beaa6087686a3211ec86881cc54066366781e8042ae73758d2ac19322/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/873087b3d7efa6ed1a424079a6b3dad163eb02301933d8f7b3a6cebfd6badded/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/48398f756eac7dd72787f61b50f3fd0db9bce7a5f942053b2214de35a501c895/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/f553d9b394743af6e23c6c2498b01992523b1300acca51cbe070ea14b431b8d0/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/ae5dc7b039400e06b6da50c881dc478fb83ed233d61914857ca782953c26d6dc/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/5ef053e0410a5a21da899372f60fe1dc5e4c69fdc1f11a4c0b866d940214c271/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/0d11b5fef8b3de9e6faee5d93643013c7f56e479f848c5e1ea5ffa9c8ca4885c/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/a30398227e479ea1fc552e3f47c63020fe2910479987585862ab36eac3b4f8e9/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/90b162a10b3a24794ad1bb510dfb045ff8083b722d058a6cb5acd09f86947df7/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/384a66de818e9d53b25e214707db6a469be651e848f544f0edf4e636b9ee3070/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/58196e6545ca928e87bccf17a76e10c5334a12a070f20a464968aca527537a1e/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/37f6aac175c68ef7327fb4d4746d7cf343cfc12f452ac7f411799fb5b0554e55/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/8a243ea905ed5b0690d4e5e64b8853951206986e7647ba08306cec748881b62a/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/e9fcb5c1fc4d4a027db5108dabded451c74527d296d438fa60c4b0416f2682ac/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/570dbc583c4523ded91d22fc160b5531a2b02e6f39bdd481eaa2fa80f35382aa/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/6bdd5267d2c4fd4601235df86b555e593a764c979b595f9402a7824c0ed9dd87/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/76650b301beedca8e3ea4ab4c0998d1f3dcfc1d91758d33baf2200126ddb26b6/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/2dc24c4961ef94961fc442971001dcc38b1ae63ff2b9c572f02da72556451b9b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/3e9893ae2fd526208e69541277761f692e371b3a87d71155c6b72b318f11cf25/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f0364d67877d772863b809e2401db4400ba5bfb3bbbf23905a185d6048e3e264/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/2aca93f6f199badaa3e416e8213ac955e2084559a8448564048f0e1fd763d6a8/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/ea7ac0cffb98af50d8a76abe75f4203573209c593429732be20c42cd47350a5b/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/bb5efe631bdb60fe2ec2376596478fa9677ab7233d6138475804c462809f5a36/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/3a72748a98bf8eb2fbe348baafce7d9fb09aa657f8641926c46d9dd5d431b57f/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/c1c43a2101ca3ec5ac39eb29b53a9f4fce02ea1933c7ad4feca9ab6aad4dd32a/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/568a2a2e442cfc18ae769b0e06b180eacfe137d98de4f1cc462bc64f36adb722/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/59e4afe16ca823cba2169c16f0e6e6759344d167def12d1e8feaec7dcf447405/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/0cf7188e7950ecc703b91a9d966a57062b69ea6b2f0498fb9b2af6522a6dd501/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/c04827d00dae6af32d0e7c99e20d48ad71a878c71c0bd9fae268f7139e274dd7/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/4b3be5c9552e98f6b757e530b71c64e8d685f8822405cfba88b2debfe62b33d4/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/a6d14665d6e1978c448e79c8fff6546d0f98a2a75ab765ef0c225d74ea31abe5/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/24df428bf2915bd8e52f88f0b932bcf21a7206c73fa940e13650d816680dc041/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/b8bbf02658a63e8adaea527538436c00e99fc4d34a355c4923f9c5b7e66656c0/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/60939e7e050f1229562508f00dceabba92aed544c05a228e0b6fbc446f1ffa94/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/4646a34ac023fd02e474a72f69605ed64a0ad2a8219dd73518302c6895a6203f/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 10:45:09.576817 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.569886 4289 manager.go:217] Machine: {Timestamp:2026-10-30 10:45:09.569419106 +0000 UTC m=+0.070798581 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106607 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28d00138eb7f563a421dcdff06e4ed SystemUUID:ec28d001-38eb-7f56-3a42-1dcdff06e4ed BootID:3139dbc3-b461-4817-b629-2ae03f4b7682 Filesystems:[{Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4b288c6-5317-410f-9f88-0352f4164d20/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9f26208e-e65a-483d-ba73-d45fe31084bc/volumes/kubernetes.io~projected/kube-api-access-4pzwb DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/545d644eb3c1bb16fdd8358db707739b2bb7321f97b3f5e75edadc84ba11168f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4b288c6-5317-410f-9f88-0352f4164d20/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/538de0b2-f736-4723-92d6-e600b9de4a26/volumes/kubernetes.io~projected/kube-api-access-xrb2t DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45d9d286-4a6a-4ebb-91ab-324335c4a2ab/volumes/kubernetes.io~projected/kube-api-access-q8pl4 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00c3d5844216d347f4d1b516b3a43e4b86b7fa9b97a7a2b78bf8b6ca66c0945a/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/63196a40-0beb-4a8b-b396-0a78f57560f3/volumes/kubernetes.io~projected/kube-api-access-58d4h DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63196a40-0beb-4a8b-b396-0a78f57560f3/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7437f570-02f5-4131-8b0b-140ed029a668/volumes/kubernetes.io~projected/kube-api-access-pv7hl DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/149f994b0fde7bb8a2e011e60cf13a1aa60718df8878ae2891567ce7dae652c6/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/08847f14edf9826f66a92064b7ca373f261cb2f09f834c0d68046b9f0f721f9b/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4b288c6-5317-410f-9f88-0352f4164d20/volumes/kubernetes.io~projected/kube-api-access-k9g8f DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/63196a40-0beb-4a8b-b396-0a78f57560f3/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5268532eb0f982280cbc1c197e09c3907f54bda7245bc79080cd9ac2fbeda6f/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c64ab741-bc4f-4006-932a-214cf53b6dc3/volumes/kubernetes.io~projected/kube-api-access-bflhl DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c5a6238e5c42ce712e1c576150adf2b35ce71c93d62fddc1a72c8a6a7b515ee/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc410460dda0dc3b4b216b489316da2ebc81a3aee74e7461260ed8ae49db6e94/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce/volumes/kubernetes.io~projected/kube-api-access-wqjpq DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f0a9655b588d3a8d23f1e9090caf5b0a8c49ac951b41013762ce64399de158a/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1389d1ff3c310f1cf00b31a6d1be34fbf370e4b033f5285d965c957418f67274/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/23abd71b-1521-4aec-9d79-198f0f8e14a2/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/275105df-bf86-436c-a464-890b4510c09d/volumes/kubernetes.io~projected/kube-api-access-rzt9k DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8/volumes/kubernetes.io~projected/kube-api-access-lp4mk DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2d110367251c32e839f6d0a99742af4fec95018d18f554439b4b7a9f54a920b/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48a8eb8a2666149b54b9c3136708009c599aba6caabd13677a5dfad1e0311aac/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d54cc91ee15ed4118ea45d0d6a1cb02b727905f2203c2637af7dc57120f9a67/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:08847f14edf9826 MacAddress:22:7d:ba:39:6a:12 Speed:10000 Mtu:8901} {Name:a5268532eb0f982 MacAddress:8e:af:2a:50:15:2a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:cb:0c:21:a3 Speed:0 Mtu:9001} {Name:br-int MacAddress:ea:62:33:35:fd:c5 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:cb:0c:21:a3 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:ca:26:e1:fd:4a:3f Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:5e:22:b4:a1:05:cd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 10:45:09.576817 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.570120 4289 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 10:45:09.576817 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.570161 4289 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 10:45:09.576817 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.570406 4289 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.570439 4289 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-242.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.570556 4289 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.570564 4289 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.570573 4289 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.570592 4289 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.570665 4289 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.570802 4289 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.571307 4289 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.571379 4289 kubelet.go:493] "Attempting to sync node with API server" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.571393 4289 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.571415 4289 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.571447 4289 kubelet.go:405] "Adding apiserver pod source" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.571459 4289 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.572253 4289 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.572265 4289 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.573780 4289 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.576532 4289 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.576546 4289 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.576675 4289 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.576938 4289 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.576984 4289 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.577096 4289 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.577113 4289 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.577122 4289 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.577131 4289 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.577141 4289 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.577150 4289 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.577159 4289 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.577168 4289 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.577178 4289 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.577188 4289 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.577203 4289 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.577218 4289 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.577258 4289 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 10:45:09.577314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.577267 4289 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 10:45:09.580889 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.580867 4289 server.go:1263] "Started kubelet" Oct 30 10:45:09.580977 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.580918 4289 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 10:45:09.581395 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.581323 4289 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 10:45:09.581376 ip-10-0-137-242 systemd[1]: Started Kubernetes Kubelet. Oct 30 10:45:09.581610 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.581406 4289 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 10:45:09.581793 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.581733 4289 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 10:45:09.582232 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.582219 4289 server.go:317] "Adding debug handlers to kubelet server" Oct 30 10:45:09.583648 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.583631 4289 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 10:45:09.583648 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.583645 4289 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 10:45:09.583855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.583678 4289 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 10:39:32 +0000 UTC" deadline="2028-06-05 14:12:16.292262822 +0000 UTC" Oct 30 10:45:09.583855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.583710 4289 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14019h27m6.708556574s" Oct 30 10:45:09.583855 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:45:09.583737 4289 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Oct 30 10:45:09.583855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.583744 4289 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 10:45:09.583855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.583756 4289 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 10:45:09.584323 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.584296 4289 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 10:45:09.585072 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585031 4289 factory.go:153] Registering CRI-O factory Oct 30 10:45:09.585072 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585048 4289 factory.go:223] Registration of the crio container factory successfully Oct 30 10:45:09.585249 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585123 4289 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 10:45:09.585249 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585134 4289 factory.go:55] Registering systemd factory Oct 30 10:45:09.585249 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585161 4289 factory.go:223] Registration of the systemd container factory successfully Oct 30 10:45:09.585249 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585190 4289 factory.go:103] Registering Raw factory Oct 30 10:45:09.585249 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585204 4289 manager.go:1199] Started watching for new ooms in manager Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585555 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="275105df-bf86-436c-a464-890b4510c09d" volumeName="kubernetes.io/configmap/275105df-bf86-436c-a464-890b4510c09d-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585589 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="275105df-bf86-436c-a464-890b4510c09d" volumeName="kubernetes.io/projected/275105df-bf86-436c-a464-890b4510c09d-kube-api-access-rzt9k" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585604 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45d9d286-4a6a-4ebb-91ab-324335c4a2ab" volumeName="kubernetes.io/projected/45d9d286-4a6a-4ebb-91ab-324335c4a2ab-kube-api-access-q8pl4" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585617 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63196a40-0beb-4a8b-b396-0a78f57560f3" volumeName="kubernetes.io/projected/63196a40-0beb-4a8b-b396-0a78f57560f3-kube-api-access-58d4h" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585630 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63196a40-0beb-4a8b-b396-0a78f57560f3" volumeName="kubernetes.io/secret/63196a40-0beb-4a8b-b396-0a78f57560f3-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585643 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7437f570-02f5-4131-8b0b-140ed029a668" volumeName="kubernetes.io/projected/7437f570-02f5-4131-8b0b-140ed029a668-kube-api-access-pv7hl" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585655 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4b288c6-5317-410f-9f88-0352f4164d20" volumeName="kubernetes.io/empty-dir/c4b288c6-5317-410f-9f88-0352f4164d20-tmp" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585667 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="275105df-bf86-436c-a464-890b4510c09d" volumeName="kubernetes.io/configmap/275105df-bf86-436c-a464-890b4510c09d-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585682 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="538de0b2-f736-4723-92d6-e600b9de4a26" volumeName="kubernetes.io/configmap/538de0b2-f736-4723-92d6-e600b9de4a26-cni-binary-copy" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585694 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="538de0b2-f736-4723-92d6-e600b9de4a26" volumeName="kubernetes.io/projected/538de0b2-f736-4723-92d6-e600b9de4a26-kube-api-access-xrb2t" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585705 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c64ab741-bc4f-4006-932a-214cf53b6dc3" volumeName="kubernetes.io/empty-dir/c64ab741-bc4f-4006-932a-214cf53b6dc3-tmp-dir" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585715 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8" volumeName="kubernetes.io/projected/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8-kube-api-access-lp4mk" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585727 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23abd71b-1521-4aec-9d79-198f0f8e14a2" volumeName="kubernetes.io/secret/23abd71b-1521-4aec-9d79-198f0f8e14a2-original-pull-secret" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585739 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45d9d286-4a6a-4ebb-91ab-324335c4a2ab" volumeName="kubernetes.io/configmap/45d9d286-4a6a-4ebb-91ab-324335c4a2ab-iptables-alerter-script" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585754 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="538de0b2-f736-4723-92d6-e600b9de4a26" volumeName="kubernetes.io/configmap/538de0b2-f736-4723-92d6-e600b9de4a26-multus-daemon-config" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585765 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7437f570-02f5-4131-8b0b-140ed029a668" volumeName="kubernetes.io/configmap/7437f570-02f5-4131-8b0b-140ed029a668-serviceca" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585777 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4b288c6-5317-410f-9f88-0352f4164d20" volumeName="kubernetes.io/empty-dir/c4b288c6-5317-410f-9f88-0352f4164d20-etc-tuned" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585790 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9" volumeName="kubernetes.io/secret/fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9-agent-certs" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585801 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63196a40-0beb-4a8b-b396-0a78f57560f3" volumeName="kubernetes.io/configmap/63196a40-0beb-4a8b-b396-0a78f57560f3-env-overrides" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585813 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63196a40-0beb-4a8b-b396-0a78f57560f3" volumeName="kubernetes.io/configmap/63196a40-0beb-4a8b-b396-0a78f57560f3-ovnkube-script-lib" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585827 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63196a40-0beb-4a8b-b396-0a78f57560f3" volumeName="kubernetes.io/configmap/63196a40-0beb-4a8b-b396-0a78f57560f3-ovnkube-config" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585841 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f26208e-e65a-483d-ba73-d45fe31084bc" volumeName="kubernetes.io/projected/9f26208e-e65a-483d-ba73-d45fe31084bc-kube-api-access-4pzwb" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585855 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b06a8e3b-6db3-4cc7-b7e2-dc14122245ce" volumeName="kubernetes.io/projected/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-kube-api-access-wqjpq" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585868 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4b288c6-5317-410f-9f88-0352f4164d20" volumeName="kubernetes.io/projected/c4b288c6-5317-410f-9f88-0352f4164d20-kube-api-access-k9g8f" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585882 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c64ab741-bc4f-4006-932a-214cf53b6dc3" volumeName="kubernetes.io/projected/c64ab741-bc4f-4006-932a-214cf53b6dc3-kube-api-access-bflhl" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585894 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9" volumeName="kubernetes.io/configmap/fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9-konnectivity-ca" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585906 4289 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="275105df-bf86-436c-a464-890b4510c09d" volumeName="kubernetes.io/configmap/275105df-bf86-436c-a464-890b4510c09d-cni-binary-copy" seLinuxMountContext="" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585920 4289 reconstruct.go:97] "Volume reconstruction finished" Oct 30 10:45:09.585903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.585930 4289 reconciler.go:29] "Reconciler: start to sync state" Oct 30 10:45:09.587536 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.586268 4289 manager.go:319] Starting recovery of all containers Oct 30 10:45:09.616911 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.616864 4289 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 10:45:09.618154 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.618132 4289 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 10:45:09.618268 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.618162 4289 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 10:45:09.618268 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.618178 4289 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 10:45:09.618268 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:45:09.618209 4289 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 10:45:09.630486 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.630458 4289 generic.go:356] "Generic (PLEG): container finished" podID="d39640f4fcc35209a29f428a929973b1" containerID="7a52ac7f1460d3889e0c6831de7897cf7a04484fa3fb234174a09a536692a6c7" exitCode=0 Oct 30 10:45:09.638955 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.638460 4289 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 10:45:09.638955 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.638594 4289 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 10:45:09.644906 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.644889 4289 manager.go:324] Recovery completed Oct 30 10:45:09.646066 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.646040 4289 generic.go:356] "Generic (PLEG): container finished" podID="275105df-bf86-436c-a464-890b4510c09d" containerID="bf3eb3226aa05ffebbd61a93c15bff17e05448727d3290ee5fe02f804254f304" exitCode=0 Oct 30 10:45:09.646153 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.646066 4289 generic.go:356] "Generic (PLEG): container finished" podID="275105df-bf86-436c-a464-890b4510c09d" containerID="c1839d7b54097b6030dd266a28b7555b2e80d48128ac87b7a9e440e6b6b343e9" exitCode=0 Oct 30 10:45:09.646153 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.646080 4289 generic.go:356] "Generic (PLEG): container finished" podID="275105df-bf86-436c-a464-890b4510c09d" containerID="86fda8e65f937ca9ff90b16ad9a7c6b59dbd8c5689d1c9723b032d306755af3d" exitCode=0 Oct 30 10:45:09.646153 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.646089 4289 generic.go:356] "Generic (PLEG): container finished" podID="275105df-bf86-436c-a464-890b4510c09d" containerID="afa6a9b7c1ea513bec91abaa8837d83f171246373b0fd63676c04f65a51854c2" exitCode=0 Oct 30 10:45:09.646153 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.646098 4289 generic.go:356] "Generic (PLEG): container finished" podID="275105df-bf86-436c-a464-890b4510c09d" containerID="06e4d993055de8f1b26a5f90b2903f29091c41b4d13228ae52fcdd8612d52db5" exitCode=0 Oct 30 10:45:09.646153 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.646106 4289 generic.go:356] "Generic (PLEG): container finished" podID="275105df-bf86-436c-a464-890b4510c09d" containerID="93ca804e6a9a413976d9635ca8677cba17f9481052c90e4c3b61f6beefbd5245" exitCode=0 Oct 30 10:45:09.648743 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.648725 4289 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 10:45:09.651043 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.651026 4289 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 10:45:09.651043 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.651033 4289 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 10:45:09.658107 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.658092 4289 cpu_manager.go:226] "Starting" policy="none" Oct 30 10:45:09.658107 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.658104 4289 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 10:45:09.658247 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.658123 4289 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 10:45:09.658299 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.658261 4289 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 10:45:09.658299 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.658277 4289 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 10:45:09.658388 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.658299 4289 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 10:45:09.658388 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.658307 4289 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 10:45:09.658388 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.658315 4289 policy_none.go:50] "Start" Oct 30 10:45:09.658388 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.658327 4289 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 10:45:09.658388 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.658339 4289 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 10:45:09.658628 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.658549 4289 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 10:45:09.658628 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.658560 4289 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 10:45:09.658628 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.658568 4289 policy_none.go:44] "Start" Oct 30 10:45:09.662415 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.662393 4289 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 10:45:09.665066 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.665046 4289 manager.go:341] "Starting Device Plugin manager" Oct 30 10:45:09.665542 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:45:09.665071 4289 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 10:45:09.665603 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.665546 4289 server.go:87] "Starting device plugin registration server" Oct 30 10:45:09.665827 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.665808 4289 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 10:45:09.665827 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.665821 4289 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 10:45:09.665972 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.665889 4289 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 10:45:09.666022 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.665971 4289 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 10:45:09.666022 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.665998 4289 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 10:45:09.666022 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.666002 4289 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 10:45:09.666165 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.666045 4289 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T10:45:09.665990441Z","UUID":"df7d5e0e-54b2-409f-97a2-9e0ff563ae07","Handler":null,"Name":"","Endpoint":""} Oct 30 10:45:09.667174 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:45:09.667138 4289 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 10:45:09.667461 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.667440 4289 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 10:45:09.667548 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.667470 4289 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 10:45:09.718739 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.718666 4289 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal"] Oct 30 10:45:09.719119 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.719093 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" event={"ID":"d39640f4fcc35209a29f428a929973b1","Type":"ContainerStarted","Data":"d7a56cdbf1cdce5c93a90066a6bc40360249c0aa95ff2a2402bab2a15a1ab669"} Oct 30 10:45:09.719183 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.719127 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" event={"ID":"d39640f4fcc35209a29f428a929973b1","Type":"ContainerDied","Data":"7a52ac7f1460d3889e0c6831de7897cf7a04484fa3fb234174a09a536692a6c7"} Oct 30 10:45:09.719183 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.719146 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" event={"ID":"d39640f4fcc35209a29f428a929973b1","Type":"ContainerStarted","Data":"2f0a9655b588d3a8d23f1e9090caf5b0a8c49ac951b41013762ce64399de158a"} Oct 30 10:45:09.719275 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.719197 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" event={"ID":"6dc9a86d1bbc0350320467df988d567a","Type":"ContainerStarted","Data":"5841a65763d434d3306679f75b96c7b82719f99081c043a638431211ebb7c16d"} Oct 30 10:45:09.719275 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.719209 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" event={"ID":"6dc9a86d1bbc0350320467df988d567a","Type":"ContainerStarted","Data":"0d54cc91ee15ed4118ea45d0d6a1cb02b727905f2203c2637af7dc57120f9a67"} Oct 30 10:45:09.719378 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.719350 4289 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Oct 30 10:45:09.719423 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.719384 4289 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Oct 30 10:45:09.747265 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.747239 4289 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 10:45:09.747354 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:45:09.747283 4289 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-242.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Oct 30 10:45:09.747667 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.747655 4289 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 10:45:09.747733 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:45:09.747691 4289 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Oct 30 10:45:09.766733 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.766714 4289 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 10:45:09.767825 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.767811 4289 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientMemory" Oct 30 10:45:09.767902 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.767841 4289 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 10:45:09.767902 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.767856 4289 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientPID" Oct 30 10:45:09.767902 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.767890 4289 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-242.ec2.internal" Oct 30 10:45:09.801565 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.801543 4289 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-242.ec2.internal" Oct 30 10:45:09.801634 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.801601 4289 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-242.ec2.internal" Oct 30 10:45:09.840231 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.840207 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6dc9a86d1bbc0350320467df988d567a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-242.ec2.internal\" (UID: \"6dc9a86d1bbc0350320467df988d567a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Oct 30 10:45:09.840342 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.840232 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d39640f4fcc35209a29f428a929973b1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" (UID: \"d39640f4fcc35209a29f428a929973b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Oct 30 10:45:09.840342 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.840255 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d39640f4fcc35209a29f428a929973b1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" (UID: \"d39640f4fcc35209a29f428a929973b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Oct 30 10:45:09.940541 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.940509 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d39640f4fcc35209a29f428a929973b1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" (UID: \"d39640f4fcc35209a29f428a929973b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Oct 30 10:45:09.940541 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.940543 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d39640f4fcc35209a29f428a929973b1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" (UID: \"d39640f4fcc35209a29f428a929973b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Oct 30 10:45:09.940717 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.940563 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6dc9a86d1bbc0350320467df988d567a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-242.ec2.internal\" (UID: \"6dc9a86d1bbc0350320467df988d567a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Oct 30 10:45:09.940717 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.940611 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6dc9a86d1bbc0350320467df988d567a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-242.ec2.internal\" (UID: \"6dc9a86d1bbc0350320467df988d567a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Oct 30 10:45:09.940717 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.940626 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d39640f4fcc35209a29f428a929973b1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" (UID: \"d39640f4fcc35209a29f428a929973b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Oct 30 10:45:09.940717 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:09.940630 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d39640f4fcc35209a29f428a929973b1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" (UID: \"d39640f4fcc35209a29f428a929973b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Oct 30 10:45:10.573020 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.572984 4289 apiserver.go:52] "Watching apiserver" Oct 30 10:45:10.601075 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.601049 4289 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 10:45:10.601398 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.601376 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-98jtw","openshift-dns/node-resolver-gll67","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal","openshift-network-diagnostics/network-check-target-fk5zl","kube-system/konnectivity-agent-8xmhg","kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq","openshift-cluster-node-tuning-operator/tuned-rpqq9","openshift-multus/network-metrics-daemon-4cgfp","openshift-multus/multus-ftv4s","openshift-network-operator/iptables-alerter-vd8gp","openshift-ovn-kubernetes/ovnkube-node-skmk7","kube-system/global-pull-secret-syncer-vskj5","openshift-image-registry/node-ca-tvrzn"] Oct 30 10:45:10.605333 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.605299 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:45:10.619366 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.619337 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 10:45:10.619366 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.619352 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 10:45:10.619627 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.619530 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 10:45:10.619627 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.619608 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 10:45:10.619724 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.619628 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5fdc6\"" Oct 30 10:45:10.619724 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.619655 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 10:45:10.619824 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.619804 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 10:45:10.619868 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.619835 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-55bss\"" Oct 30 10:45:10.619916 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.619865 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-472bd\"" Oct 30 10:45:10.619916 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.619891 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xt7nj\"" Oct 30 10:45:10.620003 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.619980 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 10:45:10.629601 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.629583 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 10:45:10.629782 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.629758 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 10:45:10.630021 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630007 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c9pg5\"" Oct 30 10:45:10.630021 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630010 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 10:45:10.630214 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630195 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 10:45:10.630273 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630204 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-mhmcf\"" Oct 30 10:45:10.630323 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630275 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s5t5b\"" Oct 30 10:45:10.630323 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630285 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 10:45:10.630410 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630293 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vmnnx\"" Oct 30 10:45:10.630410 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630307 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 10:45:10.630410 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630342 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 10:45:10.630410 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630348 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 10:45:10.630410 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630344 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 10:45:10.630410 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630360 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 10:45:10.630410 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630324 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 10:45:10.630410 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630374 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lntjm\"" Oct 30 10:45:10.630410 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630368 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vwt65\"" Oct 30 10:45:10.630410 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630307 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 10:45:10.630410 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630397 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 10:45:10.630410 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630406 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 10:45:10.630835 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630440 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 10:45:10.630835 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630447 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 10:45:10.630835 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630460 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 10:45:10.630835 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630481 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9pzxh\"" Oct 30 10:45:10.630835 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630486 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 10:45:10.630835 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630501 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 10:45:10.630835 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630538 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 10:45:10.630835 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630550 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ghh8s\"" Oct 30 10:45:10.630835 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630580 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 10:45:10.630835 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630586 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 10:45:10.630835 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.630595 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 10:45:10.631125 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.631076 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 10:45:10.649287 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.649273 4289 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Oct 30 10:45:10.649441 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.649414 4289 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Oct 30 10:45:10.671945 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.671926 4289 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 10:45:10.672024 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:45:10.671965 4289 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-242.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Oct 30 10:45:10.672518 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.672505 4289 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 10:45:10.672565 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:45:10.672533 4289 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Oct 30 10:45:10.685471 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.685453 4289 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 10:45:10.744005 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.743986 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/275105df-bf86-436c-a464-890b4510c09d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:45:10.744114 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744016 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45d9d286-4a6a-4ebb-91ab-324335c4a2ab-host-slash\") pod \"iptables-alerter-vd8gp\" (UID: \"45d9d286-4a6a-4ebb-91ab-324335c4a2ab\") " pod="openshift-network-operator/iptables-alerter-vd8gp" Oct 30 10:45:10.744114 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744038 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-cni-netd\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.744114 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744098 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63196a40-0beb-4a8b-b396-0a78f57560f3-ovn-node-metrics-cert\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.744271 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744128 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs\") pod \"network-metrics-daemon-4cgfp\" (UID: \"b06a8e3b-6db3-4cc7-b7e2-dc14122245ce\") " pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:45:10.744271 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744153 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-sys-fs\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:45:10.744271 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744171 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7437f570-02f5-4131-8b0b-140ed029a668-serviceca\") pod \"node-ca-tvrzn\" (UID: \"7437f570-02f5-4131-8b0b-140ed029a668\") " pod="openshift-image-registry/node-ca-tvrzn" Oct 30 10:45:10.744271 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744190 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/45d9d286-4a6a-4ebb-91ab-324335c4a2ab-iptables-alerter-script\") pod \"iptables-alerter-vd8gp\" (UID: \"45d9d286-4a6a-4ebb-91ab-324335c4a2ab\") " pod="openshift-network-operator/iptables-alerter-vd8gp" Oct 30 10:45:10.744271 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744210 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-slash\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.744271 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744262 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-modprobe-d\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.744587 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744311 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c64ab741-bc4f-4006-932a-214cf53b6dc3-hosts-file\") pod \"node-resolver-gll67\" (UID: \"c64ab741-bc4f-4006-932a-214cf53b6dc3\") " pod="openshift-dns/node-resolver-gll67" Oct 30 10:45:10.744587 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744358 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9-konnectivity-ca\") pod \"konnectivity-agent-8xmhg\" (UID: \"fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9\") " pod="kube-system/konnectivity-agent-8xmhg" Oct 30 10:45:10.744587 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744496 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7437f570-02f5-4131-8b0b-140ed029a668-serviceca\") pod \"node-ca-tvrzn\" (UID: \"7437f570-02f5-4131-8b0b-140ed029a668\") " pod="openshift-image-registry/node-ca-tvrzn" Oct 30 10:45:10.744587 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744535 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/45d9d286-4a6a-4ebb-91ab-324335c4a2ab-iptables-alerter-script\") pod \"iptables-alerter-vd8gp\" (UID: \"45d9d286-4a6a-4ebb-91ab-324335c4a2ab\") " pod="openshift-network-operator/iptables-alerter-vd8gp" Oct 30 10:45:10.744587 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744550 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63196a40-0beb-4a8b-b396-0a78f57560f3-ovn-node-metrics-cert\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.744587 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744578 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9-konnectivity-ca\") pod \"konnectivity-agent-8xmhg\" (UID: \"fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9\") " pod="kube-system/konnectivity-agent-8xmhg" Oct 30 10:45:10.744587 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744494 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/275105df-bf86-436c-a464-890b4510c09d-cni-binary-copy\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:45:10.744846 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744623 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8pl4\" (UniqueName: \"kubernetes.io/projected/45d9d286-4a6a-4ebb-91ab-324335c4a2ab-kube-api-access-q8pl4\") pod \"iptables-alerter-vd8gp\" (UID: \"45d9d286-4a6a-4ebb-91ab-324335c4a2ab\") " pod="openshift-network-operator/iptables-alerter-vd8gp" Oct 30 10:45:10.744846 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744651 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-node-log\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.744846 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744674 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-sysctl-conf\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.744846 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744693 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/275105df-bf86-436c-a464-890b4510c09d-cni-binary-copy\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:45:10.744846 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744725 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-host\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.744846 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744808 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqjpq\" (UniqueName: \"kubernetes.io/projected/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-kube-api-access-wqjpq\") pod \"network-metrics-daemon-4cgfp\" (UID: \"b06a8e3b-6db3-4cc7-b7e2-dc14122245ce\") " pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:45:10.745083 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744880 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/275105df-bf86-436c-a464-890b4510c09d-system-cni-dir\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:45:10.745083 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744925 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/275105df-bf86-436c-a464-890b4510c09d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:45:10.745083 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.744951 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-cnibin\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.745083 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.745005 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-hostroot\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.745083 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.745049 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-registration-dir\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:45:10.745268 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.745095 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/275105df-bf86-436c-a464-890b4510c09d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:45:10.745360 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.745344 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/23abd71b-1521-4aec-9d79-198f0f8e14a2-original-pull-secret\") pod \"global-pull-secret-syncer-vskj5\" (UID: \"23abd71b-1521-4aec-9d79-198f0f8e14a2\") " pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:45:10.745502 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.745095 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/23abd71b-1521-4aec-9d79-198f0f8e14a2-original-pull-secret\") pod \"global-pull-secret-syncer-vskj5\" (UID: \"23abd71b-1521-4aec-9d79-198f0f8e14a2\") " pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:45:10.745544 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.745518 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-run-systemd\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.745544 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.745533 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-var-lib-openvswitch\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.745614 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.745552 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-sysconfig\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.745614 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.745566 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-sysctl-d\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.745614 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.745581 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-lib-modules\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.745718 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.745616 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-run-netns\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.745718 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.745641 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4b288c6-5317-410f-9f88-0352f4164d20-tmp\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.745718 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.745658 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4b288c6-5317-410f-9f88-0352f4164d20-etc-tuned\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.745718 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.745695 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/275105df-bf86-436c-a464-890b4510c09d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:45:10.745718 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.745713 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4b288c6-5317-410f-9f88-0352f4164d20-etc-tuned\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.745960 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.745747 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-system-cni-dir\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.745960 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.745746 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4b288c6-5317-410f-9f88-0352f4164d20-tmp\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.745960 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.745773 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-etc-kubernetes\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.745960 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.745812 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-kubelet\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.745960 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.745833 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/275105df-bf86-436c-a464-890b4510c09d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:45:10.745960 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.745845 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.745960 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.745883 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63196a40-0beb-4a8b-b396-0a78f57560f3-ovnkube-script-lib\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.745960 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.745906 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-kubernetes\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.746285 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.746265 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-socket-dir\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:45:10.746357 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.746342 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-cni-bin\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.746406 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.746376 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58d4h\" (UniqueName: \"kubernetes.io/projected/63196a40-0beb-4a8b-b396-0a78f57560f3-kube-api-access-58d4h\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.746406 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.746385 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63196a40-0beb-4a8b-b396-0a78f57560f3-ovnkube-script-lib\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.746513 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.746463 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64ab741-bc4f-4006-932a-214cf53b6dc3-tmp-dir\") pod \"node-resolver-gll67\" (UID: \"c64ab741-bc4f-4006-932a-214cf53b6dc3\") " pod="openshift-dns/node-resolver-gll67" Oct 30 10:45:10.746555 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.746517 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63196a40-0beb-4a8b-b396-0a78f57560f3-env-overrides\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.746555 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.746543 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-systemd\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.746641 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.746607 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-sys\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.746685 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.746638 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-run-openvswitch\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.746732 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.746682 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63196a40-0beb-4a8b-b396-0a78f57560f3-ovnkube-config\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.746780 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.746729 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9g8f\" (UniqueName: \"kubernetes.io/projected/c4b288c6-5317-410f-9f88-0352f4164d20-kube-api-access-k9g8f\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.746780 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.746730 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64ab741-bc4f-4006-932a-214cf53b6dc3-tmp-dir\") pod \"node-resolver-gll67\" (UID: \"c64ab741-bc4f-4006-932a-214cf53b6dc3\") " pod="openshift-dns/node-resolver-gll67" Oct 30 10:45:10.746864 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.746788 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/23abd71b-1521-4aec-9d79-198f0f8e14a2-dbus\") pod \"global-pull-secret-syncer-vskj5\" (UID: \"23abd71b-1521-4aec-9d79-198f0f8e14a2\") " pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:45:10.746864 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.746821 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-run-k8s-cni-cncf-io\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.746946 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.746892 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp4mk\" (UniqueName: \"kubernetes.io/projected/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8-kube-api-access-lp4mk\") pod \"network-check-target-fk5zl\" (UID: \"e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8\") " pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:45:10.746946 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.746918 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-var-lib-kubelet\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.747029 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.746948 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/538de0b2-f736-4723-92d6-e600b9de4a26-multus-daemon-config\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.747029 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.746958 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63196a40-0beb-4a8b-b396-0a78f57560f3-ovnkube-config\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.747029 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.746980 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-etc-selinux\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:45:10.747156 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.747124 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-var-lib-cni-multus\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.747200 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.747163 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:45:10.747200 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.747191 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7437f570-02f5-4131-8b0b-140ed029a668-host\") pod \"node-ca-tvrzn\" (UID: \"7437f570-02f5-4131-8b0b-140ed029a668\") " pod="openshift-image-registry/node-ca-tvrzn" Oct 30 10:45:10.747284 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.747224 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-run\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.747284 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.747263 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-run-netns\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.747361 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.747343 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63196a40-0beb-4a8b-b396-0a78f57560f3-env-overrides\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.747452 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.747418 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-multus-conf-dir\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.747498 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.747475 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bflhl\" (UniqueName: \"kubernetes.io/projected/c64ab741-bc4f-4006-932a-214cf53b6dc3-kube-api-access-bflhl\") pod \"node-resolver-gll67\" (UID: \"c64ab741-bc4f-4006-932a-214cf53b6dc3\") " pod="openshift-dns/node-resolver-gll67" Oct 30 10:45:10.747547 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.747505 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pzwb\" (UniqueName: \"kubernetes.io/projected/9f26208e-e65a-483d-ba73-d45fe31084bc-kube-api-access-4pzwb\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:45:10.747547 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.747529 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv7hl\" (UniqueName: \"kubernetes.io/projected/7437f570-02f5-4131-8b0b-140ed029a668-kube-api-access-pv7hl\") pod \"node-ca-tvrzn\" (UID: \"7437f570-02f5-4131-8b0b-140ed029a668\") " pod="openshift-image-registry/node-ca-tvrzn" Oct 30 10:45:10.747632 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.747558 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-run-multus-certs\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.747632 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.747589 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-log-socket\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.747719 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.747683 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/275105df-bf86-436c-a464-890b4510c09d-cnibin\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:45:10.747761 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.747715 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzt9k\" (UniqueName: \"kubernetes.io/projected/275105df-bf86-436c-a464-890b4510c09d-kube-api-access-rzt9k\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:45:10.747761 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.747746 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-multus-cni-dir\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.747761 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.747753 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/538de0b2-f736-4723-92d6-e600b9de4a26-multus-daemon-config\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.747890 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.747788 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/538de0b2-f736-4723-92d6-e600b9de4a26-cni-binary-copy\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.747890 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.747840 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-var-lib-cni-bin\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.748092 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.748075 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/538de0b2-f736-4723-92d6-e600b9de4a26-cni-binary-copy\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.748151 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.748127 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-var-lib-kubelet\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.748855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.748834 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrb2t\" (UniqueName: \"kubernetes.io/projected/538de0b2-f736-4723-92d6-e600b9de4a26-kube-api-access-xrb2t\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.748937 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.748880 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-systemd-units\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.748982 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.748947 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-run-ovn-kubernetes\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.749026 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.748978 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-device-dir\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:45:10.749026 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.749011 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-multus-socket-dir-parent\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.749118 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.749041 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-etc-openvswitch\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.749118 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.749097 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-run-ovn\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.749201 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.749129 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/23abd71b-1521-4aec-9d79-198f0f8e14a2-kubelet-config\") pod \"global-pull-secret-syncer-vskj5\" (UID: \"23abd71b-1521-4aec-9d79-198f0f8e14a2\") " pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:45:10.749201 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.749162 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9-agent-certs\") pod \"konnectivity-agent-8xmhg\" (UID: \"fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9\") " pod="kube-system/konnectivity-agent-8xmhg" Oct 30 10:45:10.749534 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.749513 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9-agent-certs\") pod \"konnectivity-agent-8xmhg\" (UID: \"fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9\") " pod="kube-system/konnectivity-agent-8xmhg" Oct 30 10:45:10.769876 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.769853 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8pl4\" (UniqueName: \"kubernetes.io/projected/45d9d286-4a6a-4ebb-91ab-324335c4a2ab-kube-api-access-q8pl4\") pod \"iptables-alerter-vd8gp\" (UID: \"45d9d286-4a6a-4ebb-91ab-324335c4a2ab\") " pod="openshift-network-operator/iptables-alerter-vd8gp" Oct 30 10:45:10.769981 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.769960 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqjpq\" (UniqueName: \"kubernetes.io/projected/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-kube-api-access-wqjpq\") pod \"network-metrics-daemon-4cgfp\" (UID: \"b06a8e3b-6db3-4cc7-b7e2-dc14122245ce\") " pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:45:10.773416 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.773400 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzt9k\" (UniqueName: \"kubernetes.io/projected/275105df-bf86-436c-a464-890b4510c09d-kube-api-access-rzt9k\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:45:10.773468 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.773420 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pzwb\" (UniqueName: \"kubernetes.io/projected/9f26208e-e65a-483d-ba73-d45fe31084bc-kube-api-access-4pzwb\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:45:10.773954 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.773939 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv7hl\" (UniqueName: \"kubernetes.io/projected/7437f570-02f5-4131-8b0b-140ed029a668-kube-api-access-pv7hl\") pod \"node-ca-tvrzn\" (UID: \"7437f570-02f5-4131-8b0b-140ed029a668\") " pod="openshift-image-registry/node-ca-tvrzn" Oct 30 10:45:10.776000 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.775984 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9g8f\" (UniqueName: \"kubernetes.io/projected/c4b288c6-5317-410f-9f88-0352f4164d20-kube-api-access-k9g8f\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.777865 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.777851 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58d4h\" (UniqueName: \"kubernetes.io/projected/63196a40-0beb-4a8b-b396-0a78f57560f3-kube-api-access-58d4h\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.778258 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.778240 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bflhl\" (UniqueName: \"kubernetes.io/projected/c64ab741-bc4f-4006-932a-214cf53b6dc3-kube-api-access-bflhl\") pod \"node-resolver-gll67\" (UID: \"c64ab741-bc4f-4006-932a-214cf53b6dc3\") " pod="openshift-dns/node-resolver-gll67" Oct 30 10:45:10.778507 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.778486 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrb2t\" (UniqueName: \"kubernetes.io/projected/538de0b2-f736-4723-92d6-e600b9de4a26-kube-api-access-xrb2t\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.778661 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.778646 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp4mk\" (UniqueName: \"kubernetes.io/projected/e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8-kube-api-access-lp4mk\") pod \"network-check-target-fk5zl\" (UID: \"e66d9bf9-ae24-4a8c-bf21-03a35d3f2ec8\") " pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:45:10.850247 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850186 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-etc-selinux\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:45:10.850247 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850215 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-var-lib-cni-multus\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.850247 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850231 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:45:10.850415 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850252 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7437f570-02f5-4131-8b0b-140ed029a668-host\") pod \"node-ca-tvrzn\" (UID: \"7437f570-02f5-4131-8b0b-140ed029a668\") " pod="openshift-image-registry/node-ca-tvrzn" Oct 30 10:45:10.850415 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850306 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-var-lib-cni-multus\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.850415 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850327 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-run\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.850415 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850344 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-run-netns\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.850415 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850358 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-multus-conf-dir\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.850415 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850363 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7437f570-02f5-4131-8b0b-140ed029a668-host\") pod \"node-ca-tvrzn\" (UID: \"7437f570-02f5-4131-8b0b-140ed029a668\") " pod="openshift-image-registry/node-ca-tvrzn" Oct 30 10:45:10.850415 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850378 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-run-multus-certs\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.850415 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850308 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-etc-selinux\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:45:10.850415 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850308 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:45:10.850415 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850415 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-run-netns\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850422 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-run\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850448 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-run-multus-certs\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850477 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-multus-conf-dir\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850474 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-log-socket\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850515 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/275105df-bf86-436c-a464-890b4510c09d-cnibin\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850534 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-log-socket\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850543 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-multus-cni-dir\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850569 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/275105df-bf86-436c-a464-890b4510c09d-cnibin\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850569 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-var-lib-cni-bin\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850592 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-multus-cni-dir\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850605 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-var-lib-kubelet\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850625 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-systemd-units\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850639 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-run-ovn-kubernetes\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850655 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-device-dir\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850607 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-var-lib-cni-bin\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850677 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-multus-socket-dir-parent\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850701 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-systemd-units\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850700 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-etc-openvswitch\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850734 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-run-ovn\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850705 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-run-ovn-kubernetes\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850737 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-device-dir\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850744 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-etc-openvswitch\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850732 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-var-lib-kubelet\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850751 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-multus-socket-dir-parent\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850754 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/23abd71b-1521-4aec-9d79-198f0f8e14a2-kubelet-config\") pod \"global-pull-secret-syncer-vskj5\" (UID: \"23abd71b-1521-4aec-9d79-198f0f8e14a2\") " pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850775 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-run-ovn\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850808 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/23abd71b-1521-4aec-9d79-198f0f8e14a2-kubelet-config\") pod \"global-pull-secret-syncer-vskj5\" (UID: \"23abd71b-1521-4aec-9d79-198f0f8e14a2\") " pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850846 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/275105df-bf86-436c-a464-890b4510c09d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850878 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/275105df-bf86-436c-a464-890b4510c09d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850901 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45d9d286-4a6a-4ebb-91ab-324335c4a2ab-host-slash\") pod \"iptables-alerter-vd8gp\" (UID: \"45d9d286-4a6a-4ebb-91ab-324335c4a2ab\") " pod="openshift-network-operator/iptables-alerter-vd8gp" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850926 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-cni-netd\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.850922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850930 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45d9d286-4a6a-4ebb-91ab-324335c4a2ab-host-slash\") pod \"iptables-alerter-vd8gp\" (UID: \"45d9d286-4a6a-4ebb-91ab-324335c4a2ab\") " pod="openshift-network-operator/iptables-alerter-vd8gp" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850950 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs\") pod \"network-metrics-daemon-4cgfp\" (UID: \"b06a8e3b-6db3-4cc7-b7e2-dc14122245ce\") " pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850967 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-cni-netd\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.850976 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-sys-fs\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851001 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-slash\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851024 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-modprobe-d\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851052 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c64ab741-bc4f-4006-932a-214cf53b6dc3-hosts-file\") pod \"node-resolver-gll67\" (UID: \"c64ab741-bc4f-4006-932a-214cf53b6dc3\") " pod="openshift-dns/node-resolver-gll67" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851073 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-slash\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851073 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-sys-fs\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851076 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-node-log\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851108 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-node-log\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851118 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c64ab741-bc4f-4006-932a-214cf53b6dc3-hosts-file\") pod \"node-resolver-gll67\" (UID: \"c64ab741-bc4f-4006-932a-214cf53b6dc3\") " pod="openshift-dns/node-resolver-gll67" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851119 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-sysctl-conf\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851136 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-modprobe-d\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851152 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-host\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851174 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/275105df-bf86-436c-a464-890b4510c09d-system-cni-dir\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851220 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-cnibin\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851223 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-host\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851225 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-sysctl-conf\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851243 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-hostroot\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851245 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/275105df-bf86-436c-a464-890b4510c09d-system-cni-dir\") pod \"multus-additional-cni-plugins-98jtw\" (UID: \"275105df-bf86-436c-a464-890b4510c09d\") " pod="openshift-multus/multus-additional-cni-plugins-98jtw" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851273 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-hostroot\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851281 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-cnibin\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851296 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-registration-dir\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851313 4289 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851332 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-run-systemd\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851352 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-var-lib-openvswitch\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851369 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-run-systemd\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851376 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-sysconfig\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851397 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-registration-dir\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851403 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-var-lib-openvswitch\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851403 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-sysctl-d\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851454 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-lib-modules\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851453 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-sysconfig\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851479 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-run-netns\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851503 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-run-netns\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851507 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-sysctl-d\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851510 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-system-cni-dir\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851540 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-lib-modules\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851540 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-etc-kubernetes\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851553 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-system-cni-dir\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851571 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-etc-kubernetes\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851574 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-kubelet\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851596 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851624 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851630 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-kubernetes\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851650 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-socket-dir\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851656 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-kubelet\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851682 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-kubernetes\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851689 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-cni-bin\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851692 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9f26208e-e65a-483d-ba73-d45fe31084bc-socket-dir\") pod \"aws-ebs-csi-driver-node-7fkgq\" (UID: \"9f26208e-e65a-483d-ba73-d45fe31084bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fkgq" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851714 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-systemd\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851735 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-sys\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851755 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-run-openvswitch\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851785 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/23abd71b-1521-4aec-9d79-198f0f8e14a2-dbus\") pod \"global-pull-secret-syncer-vskj5\" (UID: \"23abd71b-1521-4aec-9d79-198f0f8e14a2\") " pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851803 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-etc-systemd\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851811 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-run-k8s-cni-cncf-io\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851819 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-sys\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851839 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-var-lib-kubelet\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851842 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-host-cni-bin\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851904 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63196a40-0beb-4a8b-b396-0a78f57560f3-run-openvswitch\") pod \"ovnkube-node-skmk7\" (UID: \"63196a40-0beb-4a8b-b396-0a78f57560f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851919 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/23abd71b-1521-4aec-9d79-198f0f8e14a2-dbus\") pod \"global-pull-secret-syncer-vskj5\" (UID: \"23abd71b-1521-4aec-9d79-198f0f8e14a2\") " pod="kube-system/global-pull-secret-syncer-vskj5" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851930 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/538de0b2-f736-4723-92d6-e600b9de4a26-host-run-k8s-cni-cncf-io\") pod \"multus-ftv4s\" (UID: \"538de0b2-f736-4723-92d6-e600b9de4a26\") " pod="openshift-multus/multus-ftv4s" Oct 30 10:45:10.851958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.851948 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4b288c6-5317-410f-9f88-0352f4164d20-var-lib-kubelet\") pod \"tuned-rpqq9\" (UID: \"c4b288c6-5317-410f-9f88-0352f4164d20\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqq9" Oct 30 10:45:10.853699 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.853388 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b06a8e3b-6db3-4cc7-b7e2-dc14122245ce-metrics-certs\") pod \"network-metrics-daemon-4cgfp\" (UID: \"b06a8e3b-6db3-4cc7-b7e2-dc14122245ce\") " pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:45:10.897111 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.897061 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:10.906478 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:10.906461 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4cgfp" Oct 30 10:45:11.073992 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:11.073945 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:11.220164 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:11.220095 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4cgfp"] Oct 30 10:45:11.222967 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:11.222937 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb06a8e3b_6db3_4cc7_b7e2_dc14122245ce.slice/crio-bac48ce6670a99cff0dd0fd90f10bc377493174275d4d9216be77413c0a34489 WatchSource:0}: Error finding container bac48ce6670a99cff0dd0fd90f10bc377493174275d4d9216be77413c0a34489: Status 404 returned error can't find the container with id bac48ce6670a99cff0dd0fd90f10bc377493174275d4d9216be77413c0a34489 Oct 30 10:45:11.225033 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:11.225016 4289 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 10:45:11.235460 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:11.235421 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:11.541650 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:11.541619 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:11.652882 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:11.652846 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4cgfp" event={"ID":"b06a8e3b-6db3-4cc7-b7e2-dc14122245ce","Type":"ContainerStarted","Data":"bac48ce6670a99cff0dd0fd90f10bc377493174275d4d9216be77413c0a34489"} Oct 30 10:45:11.724771 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:11.724737 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:11.892992 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:11.892905 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:12.061091 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:12.061051 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:12.329761 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:12.329717 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:12.493330 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:12.493290 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:12.656775 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:12.656742 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4cgfp" event={"ID":"b06a8e3b-6db3-4cc7-b7e2-dc14122245ce","Type":"ContainerStarted","Data":"74c49373630f5c3c93ffb2e98e2ef9ec8ba28aa6de22771781584dba33583201"} Oct 30 10:45:13.128175 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:13.128137 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:13.525174 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:13.525141 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:13.660575 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:13.660543 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4cgfp" event={"ID":"b06a8e3b-6db3-4cc7-b7e2-dc14122245ce","Type":"ContainerStarted","Data":"253198bff04153d60ce31711da505b8260c8d1e91cec74ba0957972c2a17b41d"} Oct 30 10:45:13.740681 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:13.740645 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:13.918234 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:13.918143 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:14.087480 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:14.087449 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:14.186303 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:14.186236 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:14.326447 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:14.326379 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:14.378047 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:14.378020 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:14.527355 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:14.527319 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:14.662609 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:14.662584 4289 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 10:45:16.502920 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:16.502879 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:16.657613 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:16.657570 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:16.817802 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:16.817720 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:16.995337 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:16.995302 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:17.194241 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:17.194165 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:18.390475 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:18.390424 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-skmk7" Oct 30 10:45:18.488137 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:18.488101 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:18.650081 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:18.649995 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:20.449299 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.449270 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-q554f"] Oct 30 10:45:20.451171 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.451151 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xr2hd"] Oct 30 10:45:20.451291 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.451270 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q554f" Oct 30 10:45:20.454155 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.454135 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xr2hd" Oct 30 10:45:20.465692 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.465672 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gffz9"] Oct 30 10:45:20.467342 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.467327 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gffz9" Oct 30 10:45:20.475998 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.475977 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-q554f"] Oct 30 10:45:20.476499 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.476479 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xr2hd"] Oct 30 10:45:20.483848 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.483830 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 10:45:20.483948 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.483866 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 10:45:20.489061 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.489040 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gffz9"] Oct 30 10:45:20.498999 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.498974 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 10:45:20.499104 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.499072 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 10:45:20.501170 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.501152 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cn9sm"] Oct 30 10:45:20.503058 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.503032 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jbkxt"] Oct 30 10:45:20.503181 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.503162 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cn9sm" Oct 30 10:45:20.505198 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.505183 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jbkxt" Oct 30 10:45:20.525742 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.525716 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 10:45:20.525827 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.525746 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 10:45:20.525827 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.525751 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 10:45:20.525827 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.525769 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 10:45:20.529738 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.529718 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jbkxt"] Oct 30 10:45:20.530186 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.530170 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cn9sm"] Oct 30 10:45:20.555029 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.554999 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-b69cv"] Oct 30 10:45:20.556985 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.556969 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-678d79cf9b-nm8bj"] Oct 30 10:45:20.557111 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.557096 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-b69cv" Oct 30 10:45:20.558957 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.558937 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.563390 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.563371 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 10:45:20.563575 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.563552 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sssm5\"" Oct 30 10:45:20.563575 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.563558 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hgw7c\"" Oct 30 10:45:20.563752 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.563590 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4nlgm\"" Oct 30 10:45:20.563752 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.563657 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 10:45:20.564129 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.564110 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-rkczw\"" Oct 30 10:45:20.564220 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.564113 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 10:45:20.564220 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.564142 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 10:45:20.564328 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.564277 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-r88l6\"" Oct 30 10:45:20.574799 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.574781 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 10:45:20.574921 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.574863 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 10:45:20.575500 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.575481 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-b69cv"] Oct 30 10:45:20.579093 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.579073 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:20.587319 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.587298 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 10:45:20.587846 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.587826 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-678d79cf9b-nm8bj"] Oct 30 10:45:20.603961 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.603933 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45be6210-4415-4212-ab2c-dd9638dcd4aa-metrics-tls\") pod \"dns-default-jbkxt\" (UID: \"45be6210-4415-4212-ab2c-dd9638dcd4aa\") " pod="openshift-dns/dns-default-jbkxt" Oct 30 10:45:20.603961 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.603961 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6tbv\" (UniqueName: \"kubernetes.io/projected/f2851305-d7b6-4e7c-8d93-a1a515941e5a-kube-api-access-q6tbv\") pod \"migrator-66df5d496b-q554f\" (UID: \"f2851305-d7b6-4e7c-8d93-a1a515941e5a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q554f" Oct 30 10:45:20.604118 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.603993 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/45be6210-4415-4212-ab2c-dd9638dcd4aa-config-volume\") pod \"dns-default-jbkxt\" (UID: \"45be6210-4415-4212-ab2c-dd9638dcd4aa\") " pod="openshift-dns/dns-default-jbkxt" Oct 30 10:45:20.604118 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.604035 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn8fx\" (UniqueName: \"kubernetes.io/projected/45be6210-4415-4212-ab2c-dd9638dcd4aa-kube-api-access-qn8fx\") pod \"dns-default-jbkxt\" (UID: \"45be6210-4415-4212-ab2c-dd9638dcd4aa\") " pod="openshift-dns/dns-default-jbkxt" Oct 30 10:45:20.604118 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.604073 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b7b910a-9274-41cb-8d26-d4a4b269bfea-data-volume\") pod \"insights-runtime-extractor-cn9sm\" (UID: \"6b7b910a-9274-41cb-8d26-d4a4b269bfea\") " pod="openshift-insights/insights-runtime-extractor-cn9sm" Oct 30 10:45:20.604217 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.604148 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbeb8de3-2377-41cf-8fcd-5cb832935d89-cert\") pod \"ingress-canary-gffz9\" (UID: \"cbeb8de3-2377-41cf-8fcd-5cb832935d89\") " pod="openshift-ingress-canary/ingress-canary-gffz9" Oct 30 10:45:20.604217 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.604179 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b7b910a-9274-41cb-8d26-d4a4b269bfea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cn9sm\" (UID: \"6b7b910a-9274-41cb-8d26-d4a4b269bfea\") " pod="openshift-insights/insights-runtime-extractor-cn9sm" Oct 30 10:45:20.604285 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.604219 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4nfz\" (UniqueName: \"kubernetes.io/projected/cbeb8de3-2377-41cf-8fcd-5cb832935d89-kube-api-access-w4nfz\") pod \"ingress-canary-gffz9\" (UID: \"cbeb8de3-2377-41cf-8fcd-5cb832935d89\") " pod="openshift-ingress-canary/ingress-canary-gffz9" Oct 30 10:45:20.604285 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.604236 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45be6210-4415-4212-ab2c-dd9638dcd4aa-tmp-dir\") pod \"dns-default-jbkxt\" (UID: \"45be6210-4415-4212-ab2c-dd9638dcd4aa\") " pod="openshift-dns/dns-default-jbkxt" Oct 30 10:45:20.604285 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.604251 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b7b910a-9274-41cb-8d26-d4a4b269bfea-crio-socket\") pod \"insights-runtime-extractor-cn9sm\" (UID: \"6b7b910a-9274-41cb-8d26-d4a4b269bfea\") " pod="openshift-insights/insights-runtime-extractor-cn9sm" Oct 30 10:45:20.604285 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.604270 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6b609664-a123-4c86-9a09-611648959685-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-xr2hd\" (UID: \"6b609664-a123-4c86-9a09-611648959685\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xr2hd" Oct 30 10:45:20.604417 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.604288 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b7b910a-9274-41cb-8d26-d4a4b269bfea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cn9sm\" (UID: \"6b7b910a-9274-41cb-8d26-d4a4b269bfea\") " pod="openshift-insights/insights-runtime-extractor-cn9sm" Oct 30 10:45:20.604417 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.604336 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44xmw\" (UniqueName: \"kubernetes.io/projected/6b7b910a-9274-41cb-8d26-d4a4b269bfea-kube-api-access-44xmw\") pod \"insights-runtime-extractor-cn9sm\" (UID: \"6b7b910a-9274-41cb-8d26-d4a4b269bfea\") " pod="openshift-insights/insights-runtime-extractor-cn9sm" Oct 30 10:45:20.663251 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.663211 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 10:45:20.663557 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.663536 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 10:45:20.663557 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.663554 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-7hhq7\"" Oct 30 10:45:20.663669 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.663605 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gnsmw\"" Oct 30 10:45:20.663705 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.663670 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 10:45:20.705445 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.705356 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4nfz\" (UniqueName: \"kubernetes.io/projected/cbeb8de3-2377-41cf-8fcd-5cb832935d89-kube-api-access-w4nfz\") pod \"ingress-canary-gffz9\" (UID: \"cbeb8de3-2377-41cf-8fcd-5cb832935d89\") " pod="openshift-ingress-canary/ingress-canary-gffz9" Oct 30 10:45:20.705445 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.705388 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45be6210-4415-4212-ab2c-dd9638dcd4aa-tmp-dir\") pod \"dns-default-jbkxt\" (UID: \"45be6210-4415-4212-ab2c-dd9638dcd4aa\") " pod="openshift-dns/dns-default-jbkxt" Oct 30 10:45:20.705445 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.705406 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b7b910a-9274-41cb-8d26-d4a4b269bfea-crio-socket\") pod \"insights-runtime-extractor-cn9sm\" (UID: \"6b7b910a-9274-41cb-8d26-d4a4b269bfea\") " pod="openshift-insights/insights-runtime-extractor-cn9sm" Oct 30 10:45:20.705691 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.705451 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6b609664-a123-4c86-9a09-611648959685-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-xr2hd\" (UID: \"6b609664-a123-4c86-9a09-611648959685\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xr2hd" Oct 30 10:45:20.705691 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.705483 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d2541bab-5182-4b00-86ca-6d80327f2d51-registry-certificates\") pod \"image-registry-678d79cf9b-nm8bj\" (UID: \"d2541bab-5182-4b00-86ca-6d80327f2d51\") " pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.705691 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.705504 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2541bab-5182-4b00-86ca-6d80327f2d51-trusted-ca\") pod \"image-registry-678d79cf9b-nm8bj\" (UID: \"d2541bab-5182-4b00-86ca-6d80327f2d51\") " pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.705691 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.705530 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b7b910a-9274-41cb-8d26-d4a4b269bfea-crio-socket\") pod \"insights-runtime-extractor-cn9sm\" (UID: \"6b7b910a-9274-41cb-8d26-d4a4b269bfea\") " pod="openshift-insights/insights-runtime-extractor-cn9sm" Oct 30 10:45:20.705691 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.705534 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b7b910a-9274-41cb-8d26-d4a4b269bfea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cn9sm\" (UID: \"6b7b910a-9274-41cb-8d26-d4a4b269bfea\") " pod="openshift-insights/insights-runtime-extractor-cn9sm" Oct 30 10:45:20.705691 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.705672 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d2541bab-5182-4b00-86ca-6d80327f2d51-ca-trust-extracted\") pod \"image-registry-678d79cf9b-nm8bj\" (UID: \"d2541bab-5182-4b00-86ca-6d80327f2d51\") " pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.705985 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.705708 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44xmw\" (UniqueName: \"kubernetes.io/projected/6b7b910a-9274-41cb-8d26-d4a4b269bfea-kube-api-access-44xmw\") pod \"insights-runtime-extractor-cn9sm\" (UID: \"6b7b910a-9274-41cb-8d26-d4a4b269bfea\") " pod="openshift-insights/insights-runtime-extractor-cn9sm" Oct 30 10:45:20.705985 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.705751 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3de9ba8f-15d6-4745-9c27-91e6d5529648-tmp\") pod \"downloads-6dd789c8b7-b69cv\" (UID: \"3de9ba8f-15d6-4745-9c27-91e6d5529648\") " pod="openshift-console/downloads-6dd789c8b7-b69cv" Oct 30 10:45:20.705985 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.705786 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45be6210-4415-4212-ab2c-dd9638dcd4aa-tmp-dir\") pod \"dns-default-jbkxt\" (UID: \"45be6210-4415-4212-ab2c-dd9638dcd4aa\") " pod="openshift-dns/dns-default-jbkxt" Oct 30 10:45:20.705985 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.705889 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx26l\" (UniqueName: \"kubernetes.io/projected/d2541bab-5182-4b00-86ca-6d80327f2d51-kube-api-access-dx26l\") pod \"image-registry-678d79cf9b-nm8bj\" (UID: \"d2541bab-5182-4b00-86ca-6d80327f2d51\") " pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.705985 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.705919 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45be6210-4415-4212-ab2c-dd9638dcd4aa-metrics-tls\") pod \"dns-default-jbkxt\" (UID: \"45be6210-4415-4212-ab2c-dd9638dcd4aa\") " pod="openshift-dns/dns-default-jbkxt" Oct 30 10:45:20.705985 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.705956 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6tbv\" (UniqueName: \"kubernetes.io/projected/f2851305-d7b6-4e7c-8d93-a1a515941e5a-kube-api-access-q6tbv\") pod \"migrator-66df5d496b-q554f\" (UID: \"f2851305-d7b6-4e7c-8d93-a1a515941e5a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q554f" Oct 30 10:45:20.706262 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.705990 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qmq2\" (UniqueName: \"kubernetes.io/projected/3de9ba8f-15d6-4745-9c27-91e6d5529648-kube-api-access-4qmq2\") pod \"downloads-6dd789c8b7-b69cv\" (UID: \"3de9ba8f-15d6-4745-9c27-91e6d5529648\") " pod="openshift-console/downloads-6dd789c8b7-b69cv" Oct 30 10:45:20.706262 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.706100 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2541bab-5182-4b00-86ca-6d80327f2d51-registry-tls\") pod \"image-registry-678d79cf9b-nm8bj\" (UID: \"d2541bab-5182-4b00-86ca-6d80327f2d51\") " pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.706262 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.706137 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2541bab-5182-4b00-86ca-6d80327f2d51-bound-sa-token\") pod \"image-registry-678d79cf9b-nm8bj\" (UID: \"d2541bab-5182-4b00-86ca-6d80327f2d51\") " pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.706262 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.706143 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b7b910a-9274-41cb-8d26-d4a4b269bfea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cn9sm\" (UID: \"6b7b910a-9274-41cb-8d26-d4a4b269bfea\") " pod="openshift-insights/insights-runtime-extractor-cn9sm" Oct 30 10:45:20.706262 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.706250 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/45be6210-4415-4212-ab2c-dd9638dcd4aa-config-volume\") pod \"dns-default-jbkxt\" (UID: \"45be6210-4415-4212-ab2c-dd9638dcd4aa\") " pod="openshift-dns/dns-default-jbkxt" Oct 30 10:45:20.706447 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.706285 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8fx\" (UniqueName: \"kubernetes.io/projected/45be6210-4415-4212-ab2c-dd9638dcd4aa-kube-api-access-qn8fx\") pod \"dns-default-jbkxt\" (UID: \"45be6210-4415-4212-ab2c-dd9638dcd4aa\") " pod="openshift-dns/dns-default-jbkxt" Oct 30 10:45:20.706447 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.706315 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b7b910a-9274-41cb-8d26-d4a4b269bfea-data-volume\") pod \"insights-runtime-extractor-cn9sm\" (UID: \"6b7b910a-9274-41cb-8d26-d4a4b269bfea\") " pod="openshift-insights/insights-runtime-extractor-cn9sm" Oct 30 10:45:20.706556 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.706416 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d2541bab-5182-4b00-86ca-6d80327f2d51-image-registry-private-configuration\") pod \"image-registry-678d79cf9b-nm8bj\" (UID: \"d2541bab-5182-4b00-86ca-6d80327f2d51\") " pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.706556 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.706514 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbeb8de3-2377-41cf-8fcd-5cb832935d89-cert\") pod \"ingress-canary-gffz9\" (UID: \"cbeb8de3-2377-41cf-8fcd-5cb832935d89\") " pod="openshift-ingress-canary/ingress-canary-gffz9" Oct 30 10:45:20.706661 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.706552 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b7b910a-9274-41cb-8d26-d4a4b269bfea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cn9sm\" (UID: \"6b7b910a-9274-41cb-8d26-d4a4b269bfea\") " pod="openshift-insights/insights-runtime-extractor-cn9sm" Oct 30 10:45:20.706661 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.706597 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d2541bab-5182-4b00-86ca-6d80327f2d51-installation-pull-secrets\") pod \"image-registry-678d79cf9b-nm8bj\" (UID: \"d2541bab-5182-4b00-86ca-6d80327f2d51\") " pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.706732 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.706714 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b7b910a-9274-41cb-8d26-d4a4b269bfea-data-volume\") pod \"insights-runtime-extractor-cn9sm\" (UID: \"6b7b910a-9274-41cb-8d26-d4a4b269bfea\") " pod="openshift-insights/insights-runtime-extractor-cn9sm" Oct 30 10:45:20.706851 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.706826 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/45be6210-4415-4212-ab2c-dd9638dcd4aa-config-volume\") pod \"dns-default-jbkxt\" (UID: \"45be6210-4415-4212-ab2c-dd9638dcd4aa\") " pod="openshift-dns/dns-default-jbkxt" Oct 30 10:45:20.708174 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.708149 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6b609664-a123-4c86-9a09-611648959685-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-xr2hd\" (UID: \"6b609664-a123-4c86-9a09-611648959685\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xr2hd" Oct 30 10:45:20.708357 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.708337 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45be6210-4415-4212-ab2c-dd9638dcd4aa-metrics-tls\") pod \"dns-default-jbkxt\" (UID: \"45be6210-4415-4212-ab2c-dd9638dcd4aa\") " pod="openshift-dns/dns-default-jbkxt" Oct 30 10:45:20.708697 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.708680 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b7b910a-9274-41cb-8d26-d4a4b269bfea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cn9sm\" (UID: \"6b7b910a-9274-41cb-8d26-d4a4b269bfea\") " pod="openshift-insights/insights-runtime-extractor-cn9sm" Oct 30 10:45:20.708952 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.708934 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbeb8de3-2377-41cf-8fcd-5cb832935d89-cert\") pod \"ingress-canary-gffz9\" (UID: \"cbeb8de3-2377-41cf-8fcd-5cb832935d89\") " pod="openshift-ingress-canary/ingress-canary-gffz9" Oct 30 10:45:20.728220 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.728190 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44xmw\" (UniqueName: \"kubernetes.io/projected/6b7b910a-9274-41cb-8d26-d4a4b269bfea-kube-api-access-44xmw\") pod \"insights-runtime-extractor-cn9sm\" (UID: \"6b7b910a-9274-41cb-8d26-d4a4b269bfea\") " pod="openshift-insights/insights-runtime-extractor-cn9sm" Oct 30 10:45:20.728501 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.728485 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4nfz\" (UniqueName: \"kubernetes.io/projected/cbeb8de3-2377-41cf-8fcd-5cb832935d89-kube-api-access-w4nfz\") pod \"ingress-canary-gffz9\" (UID: \"cbeb8de3-2377-41cf-8fcd-5cb832935d89\") " pod="openshift-ingress-canary/ingress-canary-gffz9" Oct 30 10:45:20.728719 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.728703 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6tbv\" (UniqueName: \"kubernetes.io/projected/f2851305-d7b6-4e7c-8d93-a1a515941e5a-kube-api-access-q6tbv\") pod \"migrator-66df5d496b-q554f\" (UID: \"f2851305-d7b6-4e7c-8d93-a1a515941e5a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q554f" Oct 30 10:45:20.729037 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.729016 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn8fx\" (UniqueName: \"kubernetes.io/projected/45be6210-4415-4212-ab2c-dd9638dcd4aa-kube-api-access-qn8fx\") pod \"dns-default-jbkxt\" (UID: \"45be6210-4415-4212-ab2c-dd9638dcd4aa\") " pod="openshift-dns/dns-default-jbkxt" Oct 30 10:45:20.747950 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.747932 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:45:20.750981 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.750959 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:20.762175 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.762154 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q554f" Oct 30 10:45:20.769299 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.769279 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xr2hd" Oct 30 10:45:20.780062 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.780043 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gffz9" Oct 30 10:45:20.807762 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.807732 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qmq2\" (UniqueName: \"kubernetes.io/projected/3de9ba8f-15d6-4745-9c27-91e6d5529648-kube-api-access-4qmq2\") pod \"downloads-6dd789c8b7-b69cv\" (UID: \"3de9ba8f-15d6-4745-9c27-91e6d5529648\") " pod="openshift-console/downloads-6dd789c8b7-b69cv" Oct 30 10:45:20.807940 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.807776 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2541bab-5182-4b00-86ca-6d80327f2d51-registry-tls\") pod \"image-registry-678d79cf9b-nm8bj\" (UID: \"d2541bab-5182-4b00-86ca-6d80327f2d51\") " pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.807940 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.807803 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2541bab-5182-4b00-86ca-6d80327f2d51-bound-sa-token\") pod \"image-registry-678d79cf9b-nm8bj\" (UID: \"d2541bab-5182-4b00-86ca-6d80327f2d51\") " pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.807940 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.807846 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d2541bab-5182-4b00-86ca-6d80327f2d51-image-registry-private-configuration\") pod \"image-registry-678d79cf9b-nm8bj\" (UID: \"d2541bab-5182-4b00-86ca-6d80327f2d51\") " pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.807940 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.807875 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d2541bab-5182-4b00-86ca-6d80327f2d51-installation-pull-secrets\") pod \"image-registry-678d79cf9b-nm8bj\" (UID: \"d2541bab-5182-4b00-86ca-6d80327f2d51\") " pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.807940 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.807903 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d2541bab-5182-4b00-86ca-6d80327f2d51-registry-certificates\") pod \"image-registry-678d79cf9b-nm8bj\" (UID: \"d2541bab-5182-4b00-86ca-6d80327f2d51\") " pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.807940 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.807924 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2541bab-5182-4b00-86ca-6d80327f2d51-trusted-ca\") pod \"image-registry-678d79cf9b-nm8bj\" (UID: \"d2541bab-5182-4b00-86ca-6d80327f2d51\") " pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.808221 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.807948 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d2541bab-5182-4b00-86ca-6d80327f2d51-ca-trust-extracted\") pod \"image-registry-678d79cf9b-nm8bj\" (UID: \"d2541bab-5182-4b00-86ca-6d80327f2d51\") " pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.808221 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.807978 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3de9ba8f-15d6-4745-9c27-91e6d5529648-tmp\") pod \"downloads-6dd789c8b7-b69cv\" (UID: \"3de9ba8f-15d6-4745-9c27-91e6d5529648\") " pod="openshift-console/downloads-6dd789c8b7-b69cv" Oct 30 10:45:20.808221 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.808002 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx26l\" (UniqueName: \"kubernetes.io/projected/d2541bab-5182-4b00-86ca-6d80327f2d51-kube-api-access-dx26l\") pod \"image-registry-678d79cf9b-nm8bj\" (UID: \"d2541bab-5182-4b00-86ca-6d80327f2d51\") " pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.809964 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.809897 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d2541bab-5182-4b00-86ca-6d80327f2d51-registry-certificates\") pod \"image-registry-678d79cf9b-nm8bj\" (UID: \"d2541bab-5182-4b00-86ca-6d80327f2d51\") " pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.810222 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.810196 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3de9ba8f-15d6-4745-9c27-91e6d5529648-tmp\") pod \"downloads-6dd789c8b7-b69cv\" (UID: \"3de9ba8f-15d6-4745-9c27-91e6d5529648\") " pod="openshift-console/downloads-6dd789c8b7-b69cv" Oct 30 10:45:20.810405 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.810383 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d2541bab-5182-4b00-86ca-6d80327f2d51-ca-trust-extracted\") pod \"image-registry-678d79cf9b-nm8bj\" (UID: \"d2541bab-5182-4b00-86ca-6d80327f2d51\") " pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.811369 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.811343 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2541bab-5182-4b00-86ca-6d80327f2d51-trusted-ca\") pod \"image-registry-678d79cf9b-nm8bj\" (UID: \"d2541bab-5182-4b00-86ca-6d80327f2d51\") " pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.815738 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.815712 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cn9sm" Oct 30 10:45:20.815865 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.815840 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d2541bab-5182-4b00-86ca-6d80327f2d51-installation-pull-secrets\") pod \"image-registry-678d79cf9b-nm8bj\" (UID: \"d2541bab-5182-4b00-86ca-6d80327f2d51\") " pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.815914 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.815848 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d2541bab-5182-4b00-86ca-6d80327f2d51-image-registry-private-configuration\") pod \"image-registry-678d79cf9b-nm8bj\" (UID: \"d2541bab-5182-4b00-86ca-6d80327f2d51\") " pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.819044 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.819018 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2541bab-5182-4b00-86ca-6d80327f2d51-registry-tls\") pod \"image-registry-678d79cf9b-nm8bj\" (UID: \"d2541bab-5182-4b00-86ca-6d80327f2d51\") " pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.819152 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.819136 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jbkxt" Oct 30 10:45:20.838201 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.838171 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx26l\" (UniqueName: \"kubernetes.io/projected/d2541bab-5182-4b00-86ca-6d80327f2d51-kube-api-access-dx26l\") pod \"image-registry-678d79cf9b-nm8bj\" (UID: \"d2541bab-5182-4b00-86ca-6d80327f2d51\") " pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.838997 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.838974 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qmq2\" (UniqueName: \"kubernetes.io/projected/3de9ba8f-15d6-4745-9c27-91e6d5529648-kube-api-access-4qmq2\") pod \"downloads-6dd789c8b7-b69cv\" (UID: \"3de9ba8f-15d6-4745-9c27-91e6d5529648\") " pod="openshift-console/downloads-6dd789c8b7-b69cv" Oct 30 10:45:20.839495 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.839478 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2541bab-5182-4b00-86ca-6d80327f2d51-bound-sa-token\") pod \"image-registry-678d79cf9b-nm8bj\" (UID: \"d2541bab-5182-4b00-86ca-6d80327f2d51\") " pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.868041 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.867793 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-b69cv" Oct 30 10:45:20.874761 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.873628 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:20.919796 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:20.919757 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:21.060362 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:21.060332 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-q554f"] Oct 30 10:45:21.063123 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:21.063097 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2851305_d7b6_4e7c_8d93_a1a515941e5a.slice/crio-bc81af43076be11abfb1882df664f208a91db239217840c09a132831d47d68a3 WatchSource:0}: Error finding container bc81af43076be11abfb1882df664f208a91db239217840c09a132831d47d68a3: Status 404 returned error can't find the container with id bc81af43076be11abfb1882df664f208a91db239217840c09a132831d47d68a3 Oct 30 10:45:21.086596 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:21.086473 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xr2hd"] Oct 30 10:45:21.088927 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:21.088897 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b609664_a123_4c86_9a09_611648959685.slice/crio-b0307d9ff5b0a09031542f80eeca70e83f9ef0b63407ccbd8e20c7ba24362314 WatchSource:0}: Error finding container b0307d9ff5b0a09031542f80eeca70e83f9ef0b63407ccbd8e20c7ba24362314: Status 404 returned error can't find the container with id b0307d9ff5b0a09031542f80eeca70e83f9ef0b63407ccbd8e20c7ba24362314 Oct 30 10:45:21.096232 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:21.096210 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gffz9"] Oct 30 10:45:21.098557 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:21.098537 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbeb8de3_2377_41cf_8fcd_5cb832935d89.slice/crio-25bf1af27da178623cea1f9f86b994610059aaf3bf09d15a720df346b1cbac91 WatchSource:0}: Error finding container 25bf1af27da178623cea1f9f86b994610059aaf3bf09d15a720df346b1cbac91: Status 404 returned error can't find the container with id 25bf1af27da178623cea1f9f86b994610059aaf3bf09d15a720df346b1cbac91 Oct 30 10:45:21.146159 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:21.146103 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jbkxt"] Oct 30 10:45:21.147335 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:21.147311 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cn9sm"] Oct 30 10:45:21.148495 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:21.148473 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45be6210_4415_4212_ab2c_dd9638dcd4aa.slice/crio-89ae443d049eeb5db41244421dad6ed08bc691e0f3b47f5e5d0dc43e88bdd452 WatchSource:0}: Error finding container 89ae443d049eeb5db41244421dad6ed08bc691e0f3b47f5e5d0dc43e88bdd452: Status 404 returned error can't find the container with id 89ae443d049eeb5db41244421dad6ed08bc691e0f3b47f5e5d0dc43e88bdd452 Oct 30 10:45:21.151303 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:21.151278 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7b910a_9274_41cb_8d26_d4a4b269bfea.slice/crio-4becd869ac5ae7c662f459a02af95c4c1c70e7b44a7489cc59becd2948eeb5c2 WatchSource:0}: Error finding container 4becd869ac5ae7c662f459a02af95c4c1c70e7b44a7489cc59becd2948eeb5c2: Status 404 returned error can't find the container with id 4becd869ac5ae7c662f459a02af95c4c1c70e7b44a7489cc59becd2948eeb5c2 Oct 30 10:45:21.194645 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:21.194620 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-678d79cf9b-nm8bj"] Oct 30 10:45:21.197141 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:21.197118 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2541bab_5182_4b00_86ca_6d80327f2d51.slice/crio-d238d3cf3aefc7dea2051e205d064920b4c741499fef015a2ed7867b2fe9bebe WatchSource:0}: Error finding container d238d3cf3aefc7dea2051e205d064920b4c741499fef015a2ed7867b2fe9bebe: Status 404 returned error can't find the container with id d238d3cf3aefc7dea2051e205d064920b4c741499fef015a2ed7867b2fe9bebe Oct 30 10:45:21.197912 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:21.197889 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-b69cv"] Oct 30 10:45:21.200791 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:21.200764 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de9ba8f_15d6_4745_9c27_91e6d5529648.slice/crio-e11132fa58037c85b7c41133cbb607c05a83654544d6079739ffc902a89fe72f WatchSource:0}: Error finding container e11132fa58037c85b7c41133cbb607c05a83654544d6079739ffc902a89fe72f: Status 404 returned error can't find the container with id e11132fa58037c85b7c41133cbb607c05a83654544d6079739ffc902a89fe72f Oct 30 10:45:21.686126 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:21.686006 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q554f" event={"ID":"f2851305-d7b6-4e7c-8d93-a1a515941e5a","Type":"ContainerStarted","Data":"bc81af43076be11abfb1882df664f208a91db239217840c09a132831d47d68a3"} Oct 30 10:45:21.693716 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:21.692992 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" event={"ID":"d2541bab-5182-4b00-86ca-6d80327f2d51","Type":"ContainerStarted","Data":"df9ff49613cffd99869eb759795d2f88284fb9a1f14640911bc18c5d33a07b54"} Oct 30 10:45:21.693716 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:21.693039 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" event={"ID":"d2541bab-5182-4b00-86ca-6d80327f2d51","Type":"ContainerStarted","Data":"d238d3cf3aefc7dea2051e205d064920b4c741499fef015a2ed7867b2fe9bebe"} Oct 30 10:45:21.693716 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:21.693662 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:21.696639 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:21.696544 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cn9sm" event={"ID":"6b7b910a-9274-41cb-8d26-d4a4b269bfea","Type":"ContainerStarted","Data":"bc2d18118311bad7abace5d5e6e39acbbcd3065d00598e0f4e17831c9b59a391"} Oct 30 10:45:21.696639 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:21.696593 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cn9sm" event={"ID":"6b7b910a-9274-41cb-8d26-d4a4b269bfea","Type":"ContainerStarted","Data":"4becd869ac5ae7c662f459a02af95c4c1c70e7b44a7489cc59becd2948eeb5c2"} Oct 30 10:45:21.703454 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:21.700193 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gffz9" event={"ID":"cbeb8de3-2377-41cf-8fcd-5cb832935d89","Type":"ContainerStarted","Data":"25bf1af27da178623cea1f9f86b994610059aaf3bf09d15a720df346b1cbac91"} Oct 30 10:45:21.704669 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:21.704644 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jbkxt" event={"ID":"45be6210-4415-4212-ab2c-dd9638dcd4aa","Type":"ContainerStarted","Data":"89ae443d049eeb5db41244421dad6ed08bc691e0f3b47f5e5d0dc43e88bdd452"} Oct 30 10:45:21.710078 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:21.709999 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xr2hd" event={"ID":"6b609664-a123-4c86-9a09-611648959685","Type":"ContainerStarted","Data":"b0307d9ff5b0a09031542f80eeca70e83f9ef0b63407ccbd8e20c7ba24362314"} Oct 30 10:45:21.712612 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:21.712303 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-b69cv" event={"ID":"3de9ba8f-15d6-4745-9c27-91e6d5529648","Type":"ContainerStarted","Data":"e11132fa58037c85b7c41133cbb607c05a83654544d6079739ffc902a89fe72f"} Oct 30 10:45:22.719558 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:22.719522 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q554f" event={"ID":"f2851305-d7b6-4e7c-8d93-a1a515941e5a","Type":"ContainerStarted","Data":"77b2b1022944c6d20da9efc019ca57c553abe8ad216cd4fd238b13482620f4f7"} Oct 30 10:45:22.724201 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:22.724133 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cn9sm" event={"ID":"6b7b910a-9274-41cb-8d26-d4a4b269bfea","Type":"ContainerStarted","Data":"016738bad60bc6a06f7f402dc249221595deeb93bfce3e6baa0b75cb740d07e2"} Oct 30 10:45:22.759732 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:22.759142 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" podStartSLOduration=2.759124045 podStartE2EDuration="2.759124045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:45:21.748278928 +0000 UTC m=+12.249658413" watchObservedRunningTime="2026-10-30 10:45:22.759124045 +0000 UTC m=+13.260503531" Oct 30 10:45:22.956224 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:22.956147 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:23.173518 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:23.173469 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:23.333024 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:23.332884 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:23.734778 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:23.734022 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xr2hd" event={"ID":"6b609664-a123-4c86-9a09-611648959685","Type":"ContainerStarted","Data":"4f1ff085168484543761e4707e65f3ee7ee8cd100f00dc48915bd270f02a623c"} Oct 30 10:45:23.734778 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:23.734729 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xr2hd" Oct 30 10:45:23.741403 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:23.741356 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xr2hd" Oct 30 10:45:23.772989 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:23.772931 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q554f" podStartSLOduration=2.394621364 podStartE2EDuration="3.772913418s" totalImagesPullingTime="1.378292054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:45:22.75972051 +0000 UTC m=+13.261099993" watchObservedRunningTime="2026-10-30 10:45:23.772913418 +0000 UTC m=+14.274292905" Oct 30 10:45:23.773155 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:23.773040 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xr2hd" podStartSLOduration=2.144084833 podStartE2EDuration="3.773035023s" totalImagesPullingTime="1.62895019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:45:23.772377355 +0000 UTC m=+14.273756844" watchObservedRunningTime="2026-10-30 10:45:23.773035023 +0000 UTC m=+14.274414510" Oct 30 10:45:24.169398 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:24.169361 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:24.236367 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:24.236341 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gll67_c64ab741-bc4f-4006-932a-214cf53b6dc3/dns-node-resolver/0.log" Oct 30 10:45:24.357045 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:24.356947 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:24.631162 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:24.631133 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-678d79cf9b-nm8bj_d2541bab-5182-4b00-86ca-6d80327f2d51/registry/0.log" Oct 30 10:45:24.740164 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:24.740087 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cn9sm" event={"ID":"6b7b910a-9274-41cb-8d26-d4a4b269bfea","Type":"ContainerStarted","Data":"255cd62e0649e21779a4158b56505d35fa3690544b38dfe184747b2d8ba4fe96"} Oct 30 10:45:24.742279 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:24.742252 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gffz9" event={"ID":"cbeb8de3-2377-41cf-8fcd-5cb832935d89","Type":"ContainerStarted","Data":"c4a97ac6708969e4d0bea54c3df76d12e2f4918389981f435da2d1cf3a9c7354"} Oct 30 10:45:24.744909 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:24.744877 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jbkxt" event={"ID":"45be6210-4415-4212-ab2c-dd9638dcd4aa","Type":"ContainerStarted","Data":"de0b869667e0a85c45875678c58dfbc489b72d01ea217423df94c3034aa33e3f"} Oct 30 10:45:24.745028 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:24.744927 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jbkxt" event={"ID":"45be6210-4415-4212-ab2c-dd9638dcd4aa","Type":"ContainerStarted","Data":"fdff47351e3dd77d6067c62bbe51b8289c28a254121304e9c657e4bfc0ff64f8"} Oct 30 10:45:24.745297 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:24.745280 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jbkxt" Oct 30 10:45:24.783108 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:24.783054 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cn9sm" podStartSLOduration=2.159350918 podStartE2EDuration="4.783040109s" totalImagesPullingTime="2.623689191s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 10:45:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:45:24.782447046 +0000 UTC m=+15.283826523" watchObservedRunningTime="2026-10-30 10:45:24.783040109 +0000 UTC m=+15.284419588" Oct 30 10:45:24.864344 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:24.864277 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-jbkxt" podStartSLOduration=2.213995146 podStartE2EDuration="4.86426262s" totalImagesPullingTime="2.650267474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:45:24.825061867 +0000 UTC m=+15.326441387" watchObservedRunningTime="2026-10-30 10:45:24.86426262 +0000 UTC m=+15.365642095" Oct 30 10:45:24.864715 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:24.864687 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gffz9" podStartSLOduration=1.377298607 podStartE2EDuration="4.864679155s" totalImagesPullingTime="3.487380548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:45:24.863998403 +0000 UTC m=+15.365377887" watchObservedRunningTime="2026-10-30 10:45:24.864679155 +0000 UTC m=+15.366059025" Oct 30 10:45:25.636717 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:25.636680 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tvrzn_7437f570-02f5-4131-8b0b-140ed029a668/node-ca/0.log" Oct 30 10:45:25.895385 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:25.895298 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:26.060612 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.060576 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:26.232185 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.232155 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gffz9_cbeb8de3-2377-41cf-8fcd-5cb832935d89/serve-healthcheck-canary/0.log" Oct 30 10:45:26.261583 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:45:26.261546 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:45:26.580749 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.580668 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-9445dbd7c-z8gm8"] Oct 30 10:45:26.582981 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.582957 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:26.610265 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.610234 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 10:45:26.610410 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.610295 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 10:45:26.610410 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.610306 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 10:45:26.610410 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.610337 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 10:45:26.610410 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.610370 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 10:45:26.610410 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.610399 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5bpxp\"" Oct 30 10:45:26.611178 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.611158 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9445dbd7c-z8gm8"] Oct 30 10:45:26.751531 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.751493 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a45c1443-5653-45af-9b6e-5c44220be82c-oauth-serving-cert\") pod \"console-9445dbd7c-z8gm8\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:26.751725 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.751563 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a45c1443-5653-45af-9b6e-5c44220be82c-console-config\") pod \"console-9445dbd7c-z8gm8\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:26.751725 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.751617 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a45c1443-5653-45af-9b6e-5c44220be82c-tmp\") pod \"console-9445dbd7c-z8gm8\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:26.751725 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.751714 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a45c1443-5653-45af-9b6e-5c44220be82c-console-oauth-config\") pod \"console-9445dbd7c-z8gm8\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:26.751875 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.751760 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a45c1443-5653-45af-9b6e-5c44220be82c-console-serving-cert\") pod \"console-9445dbd7c-z8gm8\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:26.751875 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.751818 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a45c1443-5653-45af-9b6e-5c44220be82c-service-ca\") pod \"console-9445dbd7c-z8gm8\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:26.751954 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.751869 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rl8w\" (UniqueName: \"kubernetes.io/projected/a45c1443-5653-45af-9b6e-5c44220be82c-kube-api-access-9rl8w\") pod \"console-9445dbd7c-z8gm8\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:26.830896 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.830793 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-q554f_f2851305-d7b6-4e7c-8d93-a1a515941e5a/migrator/0.log" Oct 30 10:45:26.852442 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.852404 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a45c1443-5653-45af-9b6e-5c44220be82c-console-config\") pod \"console-9445dbd7c-z8gm8\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:26.852442 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.852454 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a45c1443-5653-45af-9b6e-5c44220be82c-tmp\") pod \"console-9445dbd7c-z8gm8\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:26.852673 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.852499 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a45c1443-5653-45af-9b6e-5c44220be82c-console-oauth-config\") pod \"console-9445dbd7c-z8gm8\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:26.852673 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.852532 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a45c1443-5653-45af-9b6e-5c44220be82c-console-serving-cert\") pod \"console-9445dbd7c-z8gm8\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:26.852673 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.852604 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a45c1443-5653-45af-9b6e-5c44220be82c-service-ca\") pod \"console-9445dbd7c-z8gm8\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:26.852673 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.852624 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rl8w\" (UniqueName: \"kubernetes.io/projected/a45c1443-5653-45af-9b6e-5c44220be82c-kube-api-access-9rl8w\") pod \"console-9445dbd7c-z8gm8\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:26.852673 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.852664 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a45c1443-5653-45af-9b6e-5c44220be82c-oauth-serving-cert\") pod \"console-9445dbd7c-z8gm8\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:26.852922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.852893 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a45c1443-5653-45af-9b6e-5c44220be82c-tmp\") pod \"console-9445dbd7c-z8gm8\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:26.853266 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.853242 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a45c1443-5653-45af-9b6e-5c44220be82c-console-config\") pod \"console-9445dbd7c-z8gm8\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:26.853479 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.853455 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a45c1443-5653-45af-9b6e-5c44220be82c-service-ca\") pod \"console-9445dbd7c-z8gm8\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:26.853902 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.853878 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a45c1443-5653-45af-9b6e-5c44220be82c-oauth-serving-cert\") pod \"console-9445dbd7c-z8gm8\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:26.856123 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.856100 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a45c1443-5653-45af-9b6e-5c44220be82c-console-oauth-config\") pod \"console-9445dbd7c-z8gm8\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:26.856222 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.856212 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a45c1443-5653-45af-9b6e-5c44220be82c-console-serving-cert\") pod \"console-9445dbd7c-z8gm8\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:26.878231 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.878206 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rl8w\" (UniqueName: \"kubernetes.io/projected/a45c1443-5653-45af-9b6e-5c44220be82c-kube-api-access-9rl8w\") pod \"console-9445dbd7c-z8gm8\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:26.895023 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.894999 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:26.952023 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:26.951977 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:27.195074 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:27.194997 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9445dbd7c-z8gm8"] Oct 30 10:45:27.199156 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:27.199113 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda45c1443_5653_45af_9b6e_5c44220be82c.slice/crio-b8e6e74bfac8f1ca8dc48f43909ae40711029bd4abcc13ae90684bfecf0b790b WatchSource:0}: Error finding container b8e6e74bfac8f1ca8dc48f43909ae40711029bd4abcc13ae90684bfecf0b790b: Status 404 returned error can't find the container with id b8e6e74bfac8f1ca8dc48f43909ae40711029bd4abcc13ae90684bfecf0b790b Oct 30 10:45:27.758861 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:27.758822 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9445dbd7c-z8gm8" event={"ID":"a45c1443-5653-45af-9b6e-5c44220be82c","Type":"ContainerStarted","Data":"b8e6e74bfac8f1ca8dc48f43909ae40711029bd4abcc13ae90684bfecf0b790b"} Oct 30 10:45:28.608987 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.608959 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:28.773563 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.773531 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-z5s49"] Oct 30 10:45:28.775853 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.775827 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5s49" Oct 30 10:45:28.787634 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.787604 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:28.795079 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.795060 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 10:45:28.804237 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.804174 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 10:45:28.807160 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.807137 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 10:45:28.808483 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.808465 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-z5s49"] Oct 30 10:45:28.808660 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.808647 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cnt4n\"" Oct 30 10:45:28.808786 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.808775 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 10:45:28.808872 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.808863 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 10:45:28.848459 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.848295 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nbxmg"] Oct 30 10:45:28.850996 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.850971 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:28.869123 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.869091 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bab94ec6-608a-4f76-aaa9-54ddffb7d195-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-z5s49\" (UID: \"bab94ec6-608a-4f76-aaa9-54ddffb7d195\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5s49" Oct 30 10:45:28.869244 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.869142 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bab94ec6-608a-4f76-aaa9-54ddffb7d195-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-z5s49\" (UID: \"bab94ec6-608a-4f76-aaa9-54ddffb7d195\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5s49" Oct 30 10:45:28.869311 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.869252 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bab94ec6-608a-4f76-aaa9-54ddffb7d195-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-z5s49\" (UID: \"bab94ec6-608a-4f76-aaa9-54ddffb7d195\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5s49" Oct 30 10:45:28.869355 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.869307 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klblr\" (UniqueName: \"kubernetes.io/projected/bab94ec6-608a-4f76-aaa9-54ddffb7d195-kube-api-access-klblr\") pod \"openshift-state-metrics-7bd7489957-z5s49\" (UID: \"bab94ec6-608a-4f76-aaa9-54ddffb7d195\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5s49" Oct 30 10:45:28.895404 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.895360 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 10:45:28.895555 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.895487 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jgh82\"" Oct 30 10:45:28.895555 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.895533 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 10:45:28.895652 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.895604 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 10:45:28.957401 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.957301 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:28.970617 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.970591 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bab94ec6-608a-4f76-aaa9-54ddffb7d195-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-z5s49\" (UID: \"bab94ec6-608a-4f76-aaa9-54ddffb7d195\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5s49" Oct 30 10:45:28.970762 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.970635 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/200291cb-1439-4e54-a820-852ce0fa98ff-node-exporter-textfile\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:28.970762 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.970667 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/200291cb-1439-4e54-a820-852ce0fa98ff-node-exporter-accelerators-collector-config\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:28.970762 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.970705 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/200291cb-1439-4e54-a820-852ce0fa98ff-sys\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:28.970762 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.970728 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/200291cb-1439-4e54-a820-852ce0fa98ff-root\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:28.970762 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.970757 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bab94ec6-608a-4f76-aaa9-54ddffb7d195-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-z5s49\" (UID: \"bab94ec6-608a-4f76-aaa9-54ddffb7d195\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5s49" Oct 30 10:45:28.971024 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.970783 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5px5q\" (UniqueName: \"kubernetes.io/projected/200291cb-1439-4e54-a820-852ce0fa98ff-kube-api-access-5px5q\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:28.971024 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.970839 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/200291cb-1439-4e54-a820-852ce0fa98ff-metrics-client-ca\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:28.971024 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.970871 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bab94ec6-608a-4f76-aaa9-54ddffb7d195-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-z5s49\" (UID: \"bab94ec6-608a-4f76-aaa9-54ddffb7d195\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5s49" Oct 30 10:45:28.971024 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.970899 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/200291cb-1439-4e54-a820-852ce0fa98ff-node-exporter-tls\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:28.971024 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.970930 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/200291cb-1439-4e54-a820-852ce0fa98ff-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:28.971024 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.970968 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klblr\" (UniqueName: \"kubernetes.io/projected/bab94ec6-608a-4f76-aaa9-54ddffb7d195-kube-api-access-klblr\") pod \"openshift-state-metrics-7bd7489957-z5s49\" (UID: \"bab94ec6-608a-4f76-aaa9-54ddffb7d195\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5s49" Oct 30 10:45:28.971024 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.971011 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/200291cb-1439-4e54-a820-852ce0fa98ff-node-exporter-wtmp\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:28.971416 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.971395 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bab94ec6-608a-4f76-aaa9-54ddffb7d195-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-z5s49\" (UID: \"bab94ec6-608a-4f76-aaa9-54ddffb7d195\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5s49" Oct 30 10:45:28.974086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.974066 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bab94ec6-608a-4f76-aaa9-54ddffb7d195-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-z5s49\" (UID: \"bab94ec6-608a-4f76-aaa9-54ddffb7d195\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5s49" Oct 30 10:45:28.974270 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:28.974251 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bab94ec6-608a-4f76-aaa9-54ddffb7d195-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-z5s49\" (UID: \"bab94ec6-608a-4f76-aaa9-54ddffb7d195\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5s49" Oct 30 10:45:29.001400 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.001376 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klblr\" (UniqueName: \"kubernetes.io/projected/bab94ec6-608a-4f76-aaa9-54ddffb7d195-kube-api-access-klblr\") pod \"openshift-state-metrics-7bd7489957-z5s49\" (UID: \"bab94ec6-608a-4f76-aaa9-54ddffb7d195\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5s49" Oct 30 10:45:29.071863 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.071827 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/200291cb-1439-4e54-a820-852ce0fa98ff-metrics-client-ca\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:29.071863 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.071870 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/200291cb-1439-4e54-a820-852ce0fa98ff-node-exporter-tls\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:29.072104 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.071890 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/200291cb-1439-4e54-a820-852ce0fa98ff-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:29.072104 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.071917 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/200291cb-1439-4e54-a820-852ce0fa98ff-node-exporter-wtmp\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:29.072104 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.071935 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/200291cb-1439-4e54-a820-852ce0fa98ff-node-exporter-textfile\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:29.072104 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.071954 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/200291cb-1439-4e54-a820-852ce0fa98ff-node-exporter-accelerators-collector-config\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:29.072104 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.071974 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/200291cb-1439-4e54-a820-852ce0fa98ff-sys\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:29.072104 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.071988 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/200291cb-1439-4e54-a820-852ce0fa98ff-root\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:29.072104 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.072003 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5px5q\" (UniqueName: \"kubernetes.io/projected/200291cb-1439-4e54-a820-852ce0fa98ff-kube-api-access-5px5q\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:29.072892 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.072865 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/200291cb-1439-4e54-a820-852ce0fa98ff-metrics-client-ca\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:29.073024 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.072914 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/200291cb-1439-4e54-a820-852ce0fa98ff-node-exporter-textfile\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:29.073444 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.073410 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/200291cb-1439-4e54-a820-852ce0fa98ff-node-exporter-accelerators-collector-config\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:29.073533 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.073491 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/200291cb-1439-4e54-a820-852ce0fa98ff-root\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:29.073608 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.073592 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/200291cb-1439-4e54-a820-852ce0fa98ff-node-exporter-wtmp\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:29.073670 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.073631 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/200291cb-1439-4e54-a820-852ce0fa98ff-sys\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:29.075982 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.075959 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/200291cb-1439-4e54-a820-852ce0fa98ff-node-exporter-tls\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:29.078040 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.077993 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/200291cb-1439-4e54-a820-852ce0fa98ff-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:29.090849 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.090828 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5s49" Oct 30 10:45:29.100076 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.100026 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5px5q\" (UniqueName: \"kubernetes.io/projected/200291cb-1439-4e54-a820-852ce0fa98ff-kube-api-access-5px5q\") pod \"node-exporter-nbxmg\" (UID: \"200291cb-1439-4e54-a820-852ce0fa98ff\") " pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:29.128236 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.128165 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:29.166721 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.166694 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nbxmg" Oct 30 10:45:29.178835 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:29.178808 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod200291cb_1439_4e54_a820_852ce0fa98ff.slice/crio-27bdbc7d6a23d9fe38a0ff529366f0715a7940c460f6b048d300eb05ca3de815 WatchSource:0}: Error finding container 27bdbc7d6a23d9fe38a0ff529366f0715a7940c460f6b048d300eb05ca3de815: Status 404 returned error can't find the container with id 27bdbc7d6a23d9fe38a0ff529366f0715a7940c460f6b048d300eb05ca3de815 Oct 30 10:45:29.306971 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.306933 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:29.405005 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.404929 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-z5s49"] Oct 30 10:45:29.408243 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:29.408213 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab94ec6_608a_4f76_aaa9_54ddffb7d195.slice/crio-75f6a7c31bedef1160473052ae9ad619d1a86c21f5460a0b3aef9e338215b0b2 WatchSource:0}: Error finding container 75f6a7c31bedef1160473052ae9ad619d1a86c21f5460a0b3aef9e338215b0b2: Status 404 returned error can't find the container with id 75f6a7c31bedef1160473052ae9ad619d1a86c21f5460a0b3aef9e338215b0b2 Oct 30 10:45:29.470997 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.470964 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:29.768239 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.768203 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nbxmg" event={"ID":"200291cb-1439-4e54-a820-852ce0fa98ff","Type":"ContainerStarted","Data":"27bdbc7d6a23d9fe38a0ff529366f0715a7940c460f6b048d300eb05ca3de815"} Oct 30 10:45:29.771107 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.771048 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5s49" event={"ID":"bab94ec6-608a-4f76-aaa9-54ddffb7d195","Type":"ContainerStarted","Data":"a5254065d3ea15c4e431bdec829aa69d8fc7a04a6d398b6ee716dfb79070e2d0"} Oct 30 10:45:29.771107 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.771103 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5s49" event={"ID":"bab94ec6-608a-4f76-aaa9-54ddffb7d195","Type":"ContainerStarted","Data":"f065f25ddb0cdce11b10bfe2949288befcbecd9f9d7114aee7e083ff55bf597b"} Oct 30 10:45:29.771287 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.771142 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5s49" event={"ID":"bab94ec6-608a-4f76-aaa9-54ddffb7d195","Type":"ContainerStarted","Data":"75f6a7c31bedef1160473052ae9ad619d1a86c21f5460a0b3aef9e338215b0b2"} Oct 30 10:45:29.957860 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:29.957740 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:30.126831 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:30.126698 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:30.295006 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:30.294969 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:30.504158 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:30.504127 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:30.779352 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:30.779309 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9445dbd7c-z8gm8" event={"ID":"a45c1443-5653-45af-9b6e-5c44220be82c","Type":"ContainerStarted","Data":"1fe58b76999efa70dab7b38853a923c2dda9dfa4d5bfaf7e4dab2cbac6895876"} Oct 30 10:45:30.781349 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:30.781318 4289 generic.go:356] "Generic (PLEG): container finished" podID="200291cb-1439-4e54-a820-852ce0fa98ff" containerID="a8248a78804bfc3a2107833b6c0b262c41ed4b9d64dc3bed1fc5687189f4caf7" exitCode=0 Oct 30 10:45:30.781499 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:30.781398 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nbxmg" event={"ID":"200291cb-1439-4e54-a820-852ce0fa98ff","Type":"ContainerDied","Data":"a8248a78804bfc3a2107833b6c0b262c41ed4b9d64dc3bed1fc5687189f4caf7"} Oct 30 10:45:30.783722 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:30.783693 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5s49" event={"ID":"bab94ec6-608a-4f76-aaa9-54ddffb7d195","Type":"ContainerStarted","Data":"fa3743b858b8be81d7de6ec78f498b0486691dd11b3cdec324528c811ed312b1"} Oct 30 10:45:30.821981 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:30.821937 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-9445dbd7c-z8gm8" podStartSLOduration=1.7044272870000001 podStartE2EDuration="4.821922354s" totalImagesPullingTime="3.117495067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:45:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:45:30.82183967 +0000 UTC m=+21.323219156" watchObservedRunningTime="2026-10-30 10:45:30.821922354 +0000 UTC m=+21.323301839" Oct 30 10:45:30.907627 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:30.907571 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5s49" podStartSLOduration=1.857782579 podStartE2EDuration="2.907553575s" totalImagesPullingTime="1.049770996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:45:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:45:30.907177202 +0000 UTC m=+21.408556690" watchObservedRunningTime="2026-10-30 10:45:30.907553575 +0000 UTC m=+21.408933061" Oct 30 10:45:30.977208 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:30.977165 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-c86bc94d6-p9wd8"] Oct 30 10:45:30.979855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:30.979693 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:30.998716 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:30.998690 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 10:45:30.998839 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:30.998766 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 10:45:30.998839 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:30.998824 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9pfqajt2kov8p\"" Oct 30 10:45:30.998839 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:30.998833 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 10:45:30.999003 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:30.998976 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 10:45:30.999056 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:30.999026 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 10:45:30.999107 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:30.999085 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-z5jnr\"" Oct 30 10:45:31.003478 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.003453 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c86bc94d6-p9wd8"] Oct 30 10:45:31.074453 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.074407 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8xmhg" Oct 30 10:45:31.089189 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.089157 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk2xl\" (UniqueName: \"kubernetes.io/projected/a9264ef6-62e6-4756-89bc-28496a118fb3-kube-api-access-pk2xl\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.089335 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.089200 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9264ef6-62e6-4756-89bc-28496a118fb3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.089335 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.089313 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a9264ef6-62e6-4756-89bc-28496a118fb3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.089414 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.089362 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9264ef6-62e6-4756-89bc-28496a118fb3-tmp\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.089414 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.089388 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9264ef6-62e6-4756-89bc-28496a118fb3-secret-grpc-tls\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.089414 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.089408 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a9264ef6-62e6-4756-89bc-28496a118fb3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.089550 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.089439 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9264ef6-62e6-4756-89bc-28496a118fb3-metrics-client-ca\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.089550 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.089463 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a9264ef6-62e6-4756-89bc-28496a118fb3-secret-thanos-querier-tls\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.089550 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.089484 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9264ef6-62e6-4756-89bc-28496a118fb3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.190132 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.190043 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9264ef6-62e6-4756-89bc-28496a118fb3-tmp\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.190132 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.190088 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9264ef6-62e6-4756-89bc-28496a118fb3-secret-grpc-tls\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.190132 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.190122 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a9264ef6-62e6-4756-89bc-28496a118fb3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.190337 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.190149 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9264ef6-62e6-4756-89bc-28496a118fb3-metrics-client-ca\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.190337 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.190177 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a9264ef6-62e6-4756-89bc-28496a118fb3-secret-thanos-querier-tls\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.190337 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.190197 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9264ef6-62e6-4756-89bc-28496a118fb3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.190337 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.190254 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk2xl\" (UniqueName: \"kubernetes.io/projected/a9264ef6-62e6-4756-89bc-28496a118fb3-kube-api-access-pk2xl\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.190337 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.190289 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9264ef6-62e6-4756-89bc-28496a118fb3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.190337 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.190320 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a9264ef6-62e6-4756-89bc-28496a118fb3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.190585 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.190527 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9264ef6-62e6-4756-89bc-28496a118fb3-tmp\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.191743 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.191703 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9264ef6-62e6-4756-89bc-28496a118fb3-metrics-client-ca\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.193160 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.193113 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a9264ef6-62e6-4756-89bc-28496a118fb3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.193388 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.193358 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9264ef6-62e6-4756-89bc-28496a118fb3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.193710 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.193691 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9264ef6-62e6-4756-89bc-28496a118fb3-secret-grpc-tls\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.194388 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.194345 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a9264ef6-62e6-4756-89bc-28496a118fb3-secret-thanos-querier-tls\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.194607 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.194585 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9264ef6-62e6-4756-89bc-28496a118fb3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.194894 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.194874 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a9264ef6-62e6-4756-89bc-28496a118fb3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.213945 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.213920 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk2xl\" (UniqueName: \"kubernetes.io/projected/a9264ef6-62e6-4756-89bc-28496a118fb3-kube-api-access-pk2xl\") pod \"thanos-querier-c86bc94d6-p9wd8\" (UID: \"a9264ef6-62e6-4756-89bc-28496a118fb3\") " pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.286741 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.286712 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:31.298652 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.298625 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:31.477888 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.477861 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:31.628138 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.628104 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c86bc94d6-p9wd8"] Oct 30 10:45:31.631064 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:31.631031 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9264ef6_62e6_4756_89bc_28496a118fb3.slice/crio-900ab94962badb0e583901b379ab0e9d1531ff7507bf11741d93f28ec73d53e7 WatchSource:0}: Error finding container 900ab94962badb0e583901b379ab0e9d1531ff7507bf11741d93f28ec73d53e7: Status 404 returned error can't find the container with id 900ab94962badb0e583901b379ab0e9d1531ff7507bf11741d93f28ec73d53e7 Oct 30 10:45:31.789561 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.789485 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nbxmg" event={"ID":"200291cb-1439-4e54-a820-852ce0fa98ff","Type":"ContainerStarted","Data":"097d352bebcec09cfeae614e2933292430ffad37f7cd60765ee472f7b50d1bf8"} Oct 30 10:45:31.789561 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.789543 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nbxmg" event={"ID":"200291cb-1439-4e54-a820-852ce0fa98ff","Type":"ContainerStarted","Data":"25d25aacfe282f80cc3396d6ca72a7380d590145c09846dbd3109ced06469086"} Oct 30 10:45:31.791193 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.791164 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" event={"ID":"a9264ef6-62e6-4756-89bc-28496a118fb3","Type":"ContainerStarted","Data":"900ab94962badb0e583901b379ab0e9d1531ff7507bf11741d93f28ec73d53e7"} Oct 30 10:45:31.833613 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:31.833555 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nbxmg" podStartSLOduration=2.605895373 podStartE2EDuration="3.833537163s" totalImagesPullingTime="1.140565974s" totalInitContainerRuntime="87.075816ms" isStatefulPod=true podCreationTimestamp="2026-10-30 10:45:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:45:31.832139138 +0000 UTC m=+22.333518622" watchObservedRunningTime="2026-10-30 10:45:31.833537163 +0000 UTC m=+22.334916651" Oct 30 10:45:32.089512 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:32.088187 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:32.263837 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:32.263744 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:33.269334 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.269297 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5dd764cc5d-b6gfh"] Oct 30 10:45:33.272591 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.272573 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:33.287489 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.287450 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 10:45:33.301403 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.299745 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 10:45:33.301403 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.299837 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 10:45:33.301403 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.301339 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5dd764cc5d-b6gfh"] Oct 30 10:45:33.387844 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.387810 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 10:45:33.388016 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.387898 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-r69x6\"" Oct 30 10:45:33.388016 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.387918 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6f8rrivpj1vp7\"" Oct 30 10:45:33.411383 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.411353 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c6205124-095e-4ba5-a65b-fe0ee9544087-secret-metrics-server-client-certs\") pod \"metrics-server-5dd764cc5d-b6gfh\" (UID: \"c6205124-095e-4ba5-a65b-fe0ee9544087\") " pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:33.411574 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.411393 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c6205124-095e-4ba5-a65b-fe0ee9544087-secret-metrics-server-tls\") pod \"metrics-server-5dd764cc5d-b6gfh\" (UID: \"c6205124-095e-4ba5-a65b-fe0ee9544087\") " pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:33.411574 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.411423 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c6205124-095e-4ba5-a65b-fe0ee9544087-client-ca-bundle\") pod \"metrics-server-5dd764cc5d-b6gfh\" (UID: \"c6205124-095e-4ba5-a65b-fe0ee9544087\") " pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:33.411574 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.411468 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzwpc\" (UniqueName: \"kubernetes.io/projected/c6205124-095e-4ba5-a65b-fe0ee9544087-kube-api-access-lzwpc\") pod \"metrics-server-5dd764cc5d-b6gfh\" (UID: \"c6205124-095e-4ba5-a65b-fe0ee9544087\") " pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:33.411743 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.411531 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c6205124-095e-4ba5-a65b-fe0ee9544087-audit-log\") pod \"metrics-server-5dd764cc5d-b6gfh\" (UID: \"c6205124-095e-4ba5-a65b-fe0ee9544087\") " pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:33.411743 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.411735 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6205124-095e-4ba5-a65b-fe0ee9544087-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5dd764cc5d-b6gfh\" (UID: \"c6205124-095e-4ba5-a65b-fe0ee9544087\") " pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:33.411879 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.411786 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c6205124-095e-4ba5-a65b-fe0ee9544087-metrics-server-audit-profiles\") pod \"metrics-server-5dd764cc5d-b6gfh\" (UID: \"c6205124-095e-4ba5-a65b-fe0ee9544087\") " pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:33.512665 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.512630 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c6205124-095e-4ba5-a65b-fe0ee9544087-audit-log\") pod \"metrics-server-5dd764cc5d-b6gfh\" (UID: \"c6205124-095e-4ba5-a65b-fe0ee9544087\") " pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:33.512849 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.512704 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6205124-095e-4ba5-a65b-fe0ee9544087-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5dd764cc5d-b6gfh\" (UID: \"c6205124-095e-4ba5-a65b-fe0ee9544087\") " pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:33.512849 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.512739 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c6205124-095e-4ba5-a65b-fe0ee9544087-metrics-server-audit-profiles\") pod \"metrics-server-5dd764cc5d-b6gfh\" (UID: \"c6205124-095e-4ba5-a65b-fe0ee9544087\") " pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:33.512849 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.512766 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c6205124-095e-4ba5-a65b-fe0ee9544087-secret-metrics-server-client-certs\") pod \"metrics-server-5dd764cc5d-b6gfh\" (UID: \"c6205124-095e-4ba5-a65b-fe0ee9544087\") " pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:33.512849 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.512789 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c6205124-095e-4ba5-a65b-fe0ee9544087-secret-metrics-server-tls\") pod \"metrics-server-5dd764cc5d-b6gfh\" (UID: \"c6205124-095e-4ba5-a65b-fe0ee9544087\") " pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:33.512849 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.512807 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c6205124-095e-4ba5-a65b-fe0ee9544087-client-ca-bundle\") pod \"metrics-server-5dd764cc5d-b6gfh\" (UID: \"c6205124-095e-4ba5-a65b-fe0ee9544087\") " pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:33.512849 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.512831 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzwpc\" (UniqueName: \"kubernetes.io/projected/c6205124-095e-4ba5-a65b-fe0ee9544087-kube-api-access-lzwpc\") pod \"metrics-server-5dd764cc5d-b6gfh\" (UID: \"c6205124-095e-4ba5-a65b-fe0ee9544087\") " pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:33.513572 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.513549 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c6205124-095e-4ba5-a65b-fe0ee9544087-audit-log\") pod \"metrics-server-5dd764cc5d-b6gfh\" (UID: \"c6205124-095e-4ba5-a65b-fe0ee9544087\") " pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:33.514228 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.514209 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6205124-095e-4ba5-a65b-fe0ee9544087-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5dd764cc5d-b6gfh\" (UID: \"c6205124-095e-4ba5-a65b-fe0ee9544087\") " pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:33.515027 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.515005 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c6205124-095e-4ba5-a65b-fe0ee9544087-metrics-server-audit-profiles\") pod \"metrics-server-5dd764cc5d-b6gfh\" (UID: \"c6205124-095e-4ba5-a65b-fe0ee9544087\") " pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:33.517948 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.517930 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c6205124-095e-4ba5-a65b-fe0ee9544087-secret-metrics-server-tls\") pod \"metrics-server-5dd764cc5d-b6gfh\" (UID: \"c6205124-095e-4ba5-a65b-fe0ee9544087\") " pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:33.518112 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.518091 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c6205124-095e-4ba5-a65b-fe0ee9544087-client-ca-bundle\") pod \"metrics-server-5dd764cc5d-b6gfh\" (UID: \"c6205124-095e-4ba5-a65b-fe0ee9544087\") " pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:33.518942 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.518917 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c6205124-095e-4ba5-a65b-fe0ee9544087-secret-metrics-server-client-certs\") pod \"metrics-server-5dd764cc5d-b6gfh\" (UID: \"c6205124-095e-4ba5-a65b-fe0ee9544087\") " pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:33.536067 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.536016 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzwpc\" (UniqueName: \"kubernetes.io/projected/c6205124-095e-4ba5-a65b-fe0ee9544087-kube-api-access-lzwpc\") pod \"metrics-server-5dd764cc5d-b6gfh\" (UID: \"c6205124-095e-4ba5-a65b-fe0ee9544087\") " pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:33.587212 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.587177 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:33.636500 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.636464 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-jxcjp"] Oct 30 10:45:33.639703 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.639680 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-jxcjp" Oct 30 10:45:33.656254 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.655751 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-psgc6\"" Oct 30 10:45:33.656254 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.655806 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 10:45:33.656874 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.656835 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-jxcjp"] Oct 30 10:45:33.665082 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.664986 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:33.801878 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.801846 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" event={"ID":"a9264ef6-62e6-4756-89bc-28496a118fb3","Type":"ContainerStarted","Data":"1e4457fe1cf2912360df2c925c6fc0fd0dc69bff199322ec389ffd91931e235e"} Oct 30 10:45:33.815116 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.815091 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5x8x\" (UniqueName: \"kubernetes.io/projected/efe69eb0-c79d-4ea8-97e6-7bc3af0600cb-kube-api-access-r5x8x\") pod \"monitoring-plugin-77f4f76594-jxcjp\" (UID: \"efe69eb0-c79d-4ea8-97e6-7bc3af0600cb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-jxcjp" Oct 30 10:45:33.815254 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.815137 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/efe69eb0-c79d-4ea8-97e6-7bc3af0600cb-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-jxcjp\" (UID: \"efe69eb0-c79d-4ea8-97e6-7bc3af0600cb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-jxcjp" Oct 30 10:45:33.884264 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.884218 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:33.913364 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.913313 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5dd764cc5d-b6gfh"] Oct 30 10:45:33.916410 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.916382 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5x8x\" (UniqueName: \"kubernetes.io/projected/efe69eb0-c79d-4ea8-97e6-7bc3af0600cb-kube-api-access-r5x8x\") pod \"monitoring-plugin-77f4f76594-jxcjp\" (UID: \"efe69eb0-c79d-4ea8-97e6-7bc3af0600cb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-jxcjp" Oct 30 10:45:33.916535 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.916451 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/efe69eb0-c79d-4ea8-97e6-7bc3af0600cb-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-jxcjp\" (UID: \"efe69eb0-c79d-4ea8-97e6-7bc3af0600cb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-jxcjp" Oct 30 10:45:33.918045 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:33.918016 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6205124_095e_4ba5_a65b_fe0ee9544087.slice/crio-eda10cb543fe99af46be99407b57103f0e2d2697f942c7b94b7133c502069290 WatchSource:0}: Error finding container eda10cb543fe99af46be99407b57103f0e2d2697f942c7b94b7133c502069290: Status 404 returned error can't find the container with id eda10cb543fe99af46be99407b57103f0e2d2697f942c7b94b7133c502069290 Oct 30 10:45:33.919086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.919061 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/efe69eb0-c79d-4ea8-97e6-7bc3af0600cb-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-jxcjp\" (UID: \"efe69eb0-c79d-4ea8-97e6-7bc3af0600cb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-jxcjp" Oct 30 10:45:33.939149 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.939129 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5x8x\" (UniqueName: \"kubernetes.io/projected/efe69eb0-c79d-4ea8-97e6-7bc3af0600cb-kube-api-access-r5x8x\") pod \"monitoring-plugin-77f4f76594-jxcjp\" (UID: \"efe69eb0-c79d-4ea8-97e6-7bc3af0600cb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-jxcjp" Oct 30 10:45:33.957680 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:33.957657 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-jxcjp" Oct 30 10:45:34.256670 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:34.256641 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-jxcjp"] Oct 30 10:45:34.259966 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:34.259928 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe69eb0_c79d_4ea8_97e6_7bc3af0600cb.slice/crio-cd06b91a9367730fc4604181116ab04b5f0847270bd7a7175721c7ac2de17027 WatchSource:0}: Error finding container cd06b91a9367730fc4604181116ab04b5f0847270bd7a7175721c7ac2de17027: Status 404 returned error can't find the container with id cd06b91a9367730fc4604181116ab04b5f0847270bd7a7175721c7ac2de17027 Oct 30 10:45:34.814351 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:34.814295 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" event={"ID":"c6205124-095e-4ba5-a65b-fe0ee9544087","Type":"ContainerStarted","Data":"eda10cb543fe99af46be99407b57103f0e2d2697f942c7b94b7133c502069290"} Oct 30 10:45:34.822604 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:34.822567 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" event={"ID":"a9264ef6-62e6-4756-89bc-28496a118fb3","Type":"ContainerStarted","Data":"8ca5e37b35255a3175c08600d4d1258b0d7a4384d9a873a624fce6839f198aab"} Oct 30 10:45:34.822733 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:34.822654 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" event={"ID":"a9264ef6-62e6-4756-89bc-28496a118fb3","Type":"ContainerStarted","Data":"e05237ab479200473b98b9f1ee87a6f4613827f83b4d4f3b7c7044c8e0346c43"} Oct 30 10:45:34.822733 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:34.822696 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" event={"ID":"a9264ef6-62e6-4756-89bc-28496a118fb3","Type":"ContainerStarted","Data":"e6c40de12d480a78bcfc0291e156e37f518d1fe519b246b4c86ab9c6a4cc4372"} Oct 30 10:45:34.822843 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:34.822739 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" event={"ID":"a9264ef6-62e6-4756-89bc-28496a118fb3","Type":"ContainerStarted","Data":"409872817fb7112212c629d98f157c92c1ddc161200937e02b943443b9389049"} Oct 30 10:45:34.823779 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:34.823751 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-jxcjp" event={"ID":"efe69eb0-c79d-4ea8-97e6-7bc3af0600cb","Type":"ContainerStarted","Data":"cd06b91a9367730fc4604181116ab04b5f0847270bd7a7175721c7ac2de17027"} Oct 30 10:45:35.734088 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:35.734053 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:35.754539 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:35.754508 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jbkxt" Oct 30 10:45:35.835493 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:35.830031 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" event={"ID":"c6205124-095e-4ba5-a65b-fe0ee9544087","Type":"ContainerStarted","Data":"23beb3d17c438f65333cce912d8df3edd948ad8b50131e3b5356a932e7019fc3"} Oct 30 10:45:35.835493 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:35.833965 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" event={"ID":"a9264ef6-62e6-4756-89bc-28496a118fb3","Type":"ContainerStarted","Data":"bb90e85f4fb14402689dfba267ac782f888c70cfce0755115a35460e19b05b86"} Oct 30 10:45:35.835493 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:35.834487 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:35.871518 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:35.871467 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" podStartSLOduration=1.279912317 podStartE2EDuration="2.871452784s" totalImagesPullingTime="1.591540467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:45:35.870793175 +0000 UTC m=+26.372172664" watchObservedRunningTime="2026-10-30 10:45:35.871452784 +0000 UTC m=+26.372832271" Oct 30 10:45:35.917859 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:35.917805 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" podStartSLOduration=3.111592123 podStartE2EDuration="5.917790678s" totalImagesPullingTime="2.806198555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:45:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:45:35.91717634 +0000 UTC m=+26.418555823" watchObservedRunningTime="2026-10-30 10:45:35.917790678 +0000 UTC m=+26.419170162" Oct 30 10:45:35.987088 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:35.987052 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:36.218954 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:36.218874 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:36.840009 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:36.839957 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-jxcjp" event={"ID":"efe69eb0-c79d-4ea8-97e6-7bc3af0600cb","Type":"ContainerStarted","Data":"d7a64702b583e2d5c733dc2aac0944746ea550e7cef6a55827caae4d13fdb578"} Oct 30 10:45:36.840487 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:36.840172 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-jxcjp" Oct 30 10:45:36.845720 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:36.845692 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-jxcjp" Oct 30 10:45:36.879267 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:36.879221 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-jxcjp" podStartSLOduration=2.24820214 podStartE2EDuration="3.879207705s" totalImagesPullingTime="1.631005565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:45:36.879038214 +0000 UTC m=+27.380417698" watchObservedRunningTime="2026-10-30 10:45:36.879207705 +0000 UTC m=+27.380587220" Oct 30 10:45:36.895182 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:36.895158 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:36.895316 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:36.895201 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:36.900497 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:36.900478 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:37.020096 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.020062 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-956f857b5-9m5kc"] Oct 30 10:45:37.073388 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.073348 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-956f857b5-9m5kc"] Oct 30 10:45:37.073582 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.073494 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.112991 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.112917 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 10:45:37.204494 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.204458 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:37.246156 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.246120 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-trusted-ca-bundle\") pod \"console-956f857b5-9m5kc\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.246316 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.246187 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlbhj\" (UniqueName: \"kubernetes.io/projected/a4dba19a-6213-4788-a2a4-c42d7f44e76c-kube-api-access-qlbhj\") pod \"console-956f857b5-9m5kc\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.246316 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.246277 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4dba19a-6213-4788-a2a4-c42d7f44e76c-console-serving-cert\") pod \"console-956f857b5-9m5kc\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.246410 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.246315 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-console-config\") pod \"console-956f857b5-9m5kc\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.246410 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.246376 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4dba19a-6213-4788-a2a4-c42d7f44e76c-console-oauth-config\") pod \"console-956f857b5-9m5kc\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.246410 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.246394 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4dba19a-6213-4788-a2a4-c42d7f44e76c-tmp\") pod \"console-956f857b5-9m5kc\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.246556 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.246438 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-service-ca\") pod \"console-956f857b5-9m5kc\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.246556 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.246498 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-oauth-serving-cert\") pod \"console-956f857b5-9m5kc\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.347240 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.347201 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-console-config\") pod \"console-956f857b5-9m5kc\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.347425 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.347256 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4dba19a-6213-4788-a2a4-c42d7f44e76c-console-oauth-config\") pod \"console-956f857b5-9m5kc\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.347425 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.347282 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4dba19a-6213-4788-a2a4-c42d7f44e76c-tmp\") pod \"console-956f857b5-9m5kc\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.347425 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.347326 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-service-ca\") pod \"console-956f857b5-9m5kc\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.347425 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.347371 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-oauth-serving-cert\") pod \"console-956f857b5-9m5kc\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.347425 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.347416 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-trusted-ca-bundle\") pod \"console-956f857b5-9m5kc\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.347691 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.347482 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlbhj\" (UniqueName: \"kubernetes.io/projected/a4dba19a-6213-4788-a2a4-c42d7f44e76c-kube-api-access-qlbhj\") pod \"console-956f857b5-9m5kc\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.347691 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.347512 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4dba19a-6213-4788-a2a4-c42d7f44e76c-console-serving-cert\") pod \"console-956f857b5-9m5kc\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.347802 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.347758 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4dba19a-6213-4788-a2a4-c42d7f44e76c-tmp\") pod \"console-956f857b5-9m5kc\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.348099 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.348071 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-console-config\") pod \"console-956f857b5-9m5kc\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.348751 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.348718 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-oauth-serving-cert\") pod \"console-956f857b5-9m5kc\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.348853 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.348748 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-service-ca\") pod \"console-956f857b5-9m5kc\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.349065 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.349036 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-trusted-ca-bundle\") pod \"console-956f857b5-9m5kc\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.350359 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.350337 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4dba19a-6213-4788-a2a4-c42d7f44e76c-console-serving-cert\") pod \"console-956f857b5-9m5kc\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.350458 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.350362 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4dba19a-6213-4788-a2a4-c42d7f44e76c-console-oauth-config\") pod \"console-956f857b5-9m5kc\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.370955 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.370903 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlbhj\" (UniqueName: \"kubernetes.io/projected/a4dba19a-6213-4788-a2a4-c42d7f44e76c-kube-api-access-qlbhj\") pod \"console-956f857b5-9m5kc\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.386891 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.386863 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:37.389534 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.389510 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:37.564143 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.564106 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:37.724517 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.724487 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-956f857b5-9m5kc"] Oct 30 10:45:37.728964 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:45:37.728937 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4dba19a_6213_4788_a2a4_c42d7f44e76c.slice/crio-5512ab1e3bf498ab10e458d6ba41ba00135bfc3ed53125fbd474384af5414b3f WatchSource:0}: Error finding container 5512ab1e3bf498ab10e458d6ba41ba00135bfc3ed53125fbd474384af5414b3f: Status 404 returned error can't find the container with id 5512ab1e3bf498ab10e458d6ba41ba00135bfc3ed53125fbd474384af5414b3f Oct 30 10:45:37.845419 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.845382 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-956f857b5-9m5kc" event={"ID":"a4dba19a-6213-4788-a2a4-c42d7f44e76c","Type":"ContainerStarted","Data":"268968970af34b17afe16e739bb8bcb6c2a653fed73c5a95d78ddf58e5553293"} Oct 30 10:45:37.845856 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.845468 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-956f857b5-9m5kc" event={"ID":"a4dba19a-6213-4788-a2a4-c42d7f44e76c","Type":"ContainerStarted","Data":"5512ab1e3bf498ab10e458d6ba41ba00135bfc3ed53125fbd474384af5414b3f"} Oct 30 10:45:37.850276 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.850254 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:45:37.888296 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:37.888244 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-956f857b5-9m5kc" podStartSLOduration=1.8882295500000001 podStartE2EDuration="1.88822955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:45:37.886698179 +0000 UTC m=+28.388077666" watchObservedRunningTime="2026-10-30 10:45:37.88822955 +0000 UTC m=+28.389609035" Oct 30 10:45:38.659345 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:38.659304 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:39.856712 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:39.856667 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-b69cv" event={"ID":"3de9ba8f-15d6-4745-9c27-91e6d5529648","Type":"ContainerStarted","Data":"049fbd0c93c12c693b378c1d5ef6bbb2316d85edfb4cbd3876ed41325cd49cd2"} Oct 30 10:45:39.857321 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:39.856899 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-b69cv" Oct 30 10:45:39.874887 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:39.874856 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-b69cv" Oct 30 10:45:39.920510 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:39.920457 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-b69cv" podStartSLOduration=1.763709445 podStartE2EDuration="19.920442044s" totalImagesPullingTime="18.156732599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:45:39.908094524 +0000 UTC m=+30.409474022" watchObservedRunningTime="2026-10-30 10:45:39.920442044 +0000 UTC m=+30.421821555" Oct 30 10:45:40.751235 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:40.751202 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fk5zl" Oct 30 10:45:40.956896 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:40.956863 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:41.129333 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:41.129244 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:41.848290 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:41.848261 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-c86bc94d6-p9wd8" Oct 30 10:45:43.742092 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:43.741268 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-678d79cf9b-nm8bj" Oct 30 10:45:44.024556 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:44.024471 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:44.216721 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:44.216686 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:45.571654 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:45.571623 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:45.756629 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:45.756587 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:47.390499 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:47.390458 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:47.390982 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:47.390562 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:47.395879 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:47.395854 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:47.471660 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:47.471621 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:47.644019 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:47.643943 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:47.899150 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:47.899072 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:45:47.971872 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:47.971816 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9445dbd7c-z8gm8"] Oct 30 10:45:48.230607 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:48.230573 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:48.406343 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:48.406305 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:49.416654 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:49.416622 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:53.361152 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:53.361073 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:53.556562 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:53.556533 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:53.587677 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:53.587655 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:53.587791 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:53.587706 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:45:55.270176 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:55.270133 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:55.434793 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:55.434754 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:55.897848 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:55.897812 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:56.076671 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:45:56.076630 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:45:56.256556 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:45:56.256524 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:46:10.908675 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:10.908622 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:46:11.086094 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:11.086057 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:46:12.997357 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:12.997303 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-9445dbd7c-z8gm8" podUID="a45c1443-5653-45af-9b6e-5c44220be82c" containerName="console" containerID="cri-o://1fe58b76999efa70dab7b38853a923c2dda9dfa4d5bfaf7e4dab2cbac6895876" gracePeriod=15 Oct 30 10:46:13.262410 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.262349 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:46:13.320019 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.319990 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a45c1443-5653-45af-9b6e-5c44220be82c-kube-api-access-9rl8w\" (UniqueName: \"kubernetes.io/projected/a45c1443-5653-45af-9b6e-5c44220be82c-kube-api-access-9rl8w\") pod \"a45c1443-5653-45af-9b6e-5c44220be82c\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " Oct 30 10:46:13.320168 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.320060 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a45c1443-5653-45af-9b6e-5c44220be82c-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a45c1443-5653-45af-9b6e-5c44220be82c-oauth-serving-cert\") pod \"a45c1443-5653-45af-9b6e-5c44220be82c\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " Oct 30 10:46:13.320168 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.320104 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a45c1443-5653-45af-9b6e-5c44220be82c-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a45c1443-5653-45af-9b6e-5c44220be82c-tmp\") pod \"a45c1443-5653-45af-9b6e-5c44220be82c\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " Oct 30 10:46:13.320168 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.320147 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a45c1443-5653-45af-9b6e-5c44220be82c-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a45c1443-5653-45af-9b6e-5c44220be82c-console-serving-cert\") pod \"a45c1443-5653-45af-9b6e-5c44220be82c\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " Oct 30 10:46:13.320304 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.320272 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a45c1443-5653-45af-9b6e-5c44220be82c-console-config\" (UniqueName: \"kubernetes.io/configmap/a45c1443-5653-45af-9b6e-5c44220be82c-console-config\") pod \"a45c1443-5653-45af-9b6e-5c44220be82c\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " Oct 30 10:46:13.320369 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.320353 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a45c1443-5653-45af-9b6e-5c44220be82c-service-ca\" (UniqueName: \"kubernetes.io/configmap/a45c1443-5653-45af-9b6e-5c44220be82c-service-ca\") pod \"a45c1443-5653-45af-9b6e-5c44220be82c\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " Oct 30 10:46:13.320439 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.320394 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a45c1443-5653-45af-9b6e-5c44220be82c-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a45c1443-5653-45af-9b6e-5c44220be82c-console-oauth-config\") pod \"a45c1443-5653-45af-9b6e-5c44220be82c\" (UID: \"a45c1443-5653-45af-9b6e-5c44220be82c\") " Oct 30 10:46:13.320499 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.320469 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45c1443-5653-45af-9b6e-5c44220be82c-tmp" pod "a45c1443-5653-45af-9b6e-5c44220be82c" (UID: "a45c1443-5653-45af-9b6e-5c44220be82c"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:46:13.320565 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.320534 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a45c1443-5653-45af-9b6e-5c44220be82c-oauth-serving-cert" pod "a45c1443-5653-45af-9b6e-5c44220be82c" (UID: "a45c1443-5653-45af-9b6e-5c44220be82c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:46:13.320858 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.320695 4289 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a45c1443-5653-45af-9b6e-5c44220be82c-oauth-serving-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:46:13.320858 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.320721 4289 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a45c1443-5653-45af-9b6e-5c44220be82c-tmp\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:46:13.320858 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.320820 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a45c1443-5653-45af-9b6e-5c44220be82c-console-config" pod "a45c1443-5653-45af-9b6e-5c44220be82c" (UID: "a45c1443-5653-45af-9b6e-5c44220be82c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:46:13.321029 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.320951 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a45c1443-5653-45af-9b6e-5c44220be82c-service-ca" pod "a45c1443-5653-45af-9b6e-5c44220be82c" (UID: "a45c1443-5653-45af-9b6e-5c44220be82c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:46:13.322475 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.322447 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a45c1443-5653-45af-9b6e-5c44220be82c-console-serving-cert" pod "a45c1443-5653-45af-9b6e-5c44220be82c" (UID: "a45c1443-5653-45af-9b6e-5c44220be82c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:46:13.322566 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.322494 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a45c1443-5653-45af-9b6e-5c44220be82c-kube-api-access-9rl8w" pod "a45c1443-5653-45af-9b6e-5c44220be82c" (UID: "a45c1443-5653-45af-9b6e-5c44220be82c"). InnerVolumeSpecName "kube-api-access-9rl8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:46:13.322916 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.322902 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a45c1443-5653-45af-9b6e-5c44220be82c-console-oauth-config" pod "a45c1443-5653-45af-9b6e-5c44220be82c" (UID: "a45c1443-5653-45af-9b6e-5c44220be82c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:46:13.421549 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.421524 4289 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a45c1443-5653-45af-9b6e-5c44220be82c-console-serving-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:46:13.421549 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.421545 4289 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a45c1443-5653-45af-9b6e-5c44220be82c-console-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:46:13.421549 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.421555 4289 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a45c1443-5653-45af-9b6e-5c44220be82c-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:46:13.421741 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.421563 4289 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a45c1443-5653-45af-9b6e-5c44220be82c-console-oauth-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:46:13.421741 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.421571 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rl8w\" (UniqueName: \"kubernetes.io/projected/a45c1443-5653-45af-9b6e-5c44220be82c-kube-api-access-9rl8w\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:46:13.593448 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.593357 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:46:13.597086 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.597061 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5dd764cc5d-b6gfh" Oct 30 10:46:13.983226 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.983192 4289 generic.go:356] "Generic (PLEG): container finished" podID="a45c1443-5653-45af-9b6e-5c44220be82c" containerID="1fe58b76999efa70dab7b38853a923c2dda9dfa4d5bfaf7e4dab2cbac6895876" exitCode=0 Oct 30 10:46:13.983399 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.983241 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9445dbd7c-z8gm8" Oct 30 10:46:13.983399 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.983277 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9445dbd7c-z8gm8" event={"ID":"a45c1443-5653-45af-9b6e-5c44220be82c","Type":"ContainerDied","Data":"1fe58b76999efa70dab7b38853a923c2dda9dfa4d5bfaf7e4dab2cbac6895876"} Oct 30 10:46:13.983399 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.983322 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9445dbd7c-z8gm8" event={"ID":"a45c1443-5653-45af-9b6e-5c44220be82c","Type":"ContainerDied","Data":"b8e6e74bfac8f1ca8dc48f43909ae40711029bd4abcc13ae90684bfecf0b790b"} Oct 30 10:46:13.983399 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.983344 4289 scope.go:122] "RemoveContainer" containerID="1fe58b76999efa70dab7b38853a923c2dda9dfa4d5bfaf7e4dab2cbac6895876" Oct 30 10:46:13.991286 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.991269 4289 scope.go:122] "RemoveContainer" containerID="1fe58b76999efa70dab7b38853a923c2dda9dfa4d5bfaf7e4dab2cbac6895876" Oct 30 10:46:13.991565 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:46:13.991542 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fe58b76999efa70dab7b38853a923c2dda9dfa4d5bfaf7e4dab2cbac6895876\": container with ID starting with 1fe58b76999efa70dab7b38853a923c2dda9dfa4d5bfaf7e4dab2cbac6895876 not found: ID does not exist" containerID="1fe58b76999efa70dab7b38853a923c2dda9dfa4d5bfaf7e4dab2cbac6895876" Oct 30 10:46:13.991619 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:13.991578 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fe58b76999efa70dab7b38853a923c2dda9dfa4d5bfaf7e4dab2cbac6895876"} err="failed to get container status \"1fe58b76999efa70dab7b38853a923c2dda9dfa4d5bfaf7e4dab2cbac6895876\": rpc error: code = NotFound desc = could not find container \"1fe58b76999efa70dab7b38853a923c2dda9dfa4d5bfaf7e4dab2cbac6895876\": container with ID starting with 1fe58b76999efa70dab7b38853a923c2dda9dfa4d5bfaf7e4dab2cbac6895876 not found: ID does not exist" Oct 30 10:46:14.033296 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:14.033263 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9445dbd7c-z8gm8"] Oct 30 10:46:14.045028 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:14.045004 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9445dbd7c-z8gm8"] Oct 30 10:46:15.623287 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:15.623254 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a45c1443-5653-45af-9b6e-5c44220be82c" path="/var/lib/kubelet/pods/a45c1443-5653-45af-9b6e-5c44220be82c/volumes" Oct 30 10:46:18.103825 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:18.103777 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:46:18.326636 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:18.326600 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:46:26.261057 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:46:26.261015 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:46:27.410452 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.410411 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-fd67b888d-95ckz"] Oct 30 10:46:27.410930 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.410861 4289 cpu_manager.go:414] "containerMap: removing container" podUID="a45c1443-5653-45af-9b6e-5c44220be82c" containerName="console" Oct 30 10:46:27.410930 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.410881 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a45c1443-5653-45af-9b6e-5c44220be82c" containerName="console" Oct 30 10:46:27.411040 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.410948 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="a45c1443-5653-45af-9b6e-5c44220be82c" containerName="console" Oct 30 10:46:27.412882 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.412860 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.450137 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.450109 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fd67b888d-95ckz"] Oct 30 10:46:27.537105 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.537069 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bf6f910-7936-4266-a167-ebf7c5f2e0af-console-serving-cert\") pod \"console-fd67b888d-95ckz\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.537105 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.537109 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq4bt\" (UniqueName: \"kubernetes.io/projected/2bf6f910-7936-4266-a167-ebf7c5f2e0af-kube-api-access-jq4bt\") pod \"console-fd67b888d-95ckz\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.537281 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.537202 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-console-config\") pod \"console-fd67b888d-95ckz\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.537281 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.537244 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bf6f910-7936-4266-a167-ebf7c5f2e0af-tmp\") pod \"console-fd67b888d-95ckz\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.537348 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.537286 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-service-ca\") pod \"console-fd67b888d-95ckz\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.537385 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.537372 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-oauth-serving-cert\") pod \"console-fd67b888d-95ckz\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.537417 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.537390 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bf6f910-7936-4266-a167-ebf7c5f2e0af-console-oauth-config\") pod \"console-fd67b888d-95ckz\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.537417 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.537407 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-trusted-ca-bundle\") pod \"console-fd67b888d-95ckz\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.597149 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.597123 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:46:27.637740 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.637715 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-oauth-serving-cert\") pod \"console-fd67b888d-95ckz\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.637861 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.637743 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bf6f910-7936-4266-a167-ebf7c5f2e0af-console-oauth-config\") pod \"console-fd67b888d-95ckz\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.637861 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.637769 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-trusted-ca-bundle\") pod \"console-fd67b888d-95ckz\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.637861 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.637787 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bf6f910-7936-4266-a167-ebf7c5f2e0af-console-serving-cert\") pod \"console-fd67b888d-95ckz\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.637861 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.637817 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq4bt\" (UniqueName: \"kubernetes.io/projected/2bf6f910-7936-4266-a167-ebf7c5f2e0af-kube-api-access-jq4bt\") pod \"console-fd67b888d-95ckz\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.638050 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.637879 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-console-config\") pod \"console-fd67b888d-95ckz\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.638050 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.637912 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bf6f910-7936-4266-a167-ebf7c5f2e0af-tmp\") pod \"console-fd67b888d-95ckz\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.638050 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.637948 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-service-ca\") pod \"console-fd67b888d-95ckz\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.638528 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.638498 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-oauth-serving-cert\") pod \"console-fd67b888d-95ckz\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.638528 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.638519 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bf6f910-7936-4266-a167-ebf7c5f2e0af-tmp\") pod \"console-fd67b888d-95ckz\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.638713 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.638697 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-console-config\") pod \"console-fd67b888d-95ckz\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.638865 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.638835 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-service-ca\") pod \"console-fd67b888d-95ckz\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.638942 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.638835 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-trusted-ca-bundle\") pod \"console-fd67b888d-95ckz\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.640251 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.640233 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bf6f910-7936-4266-a167-ebf7c5f2e0af-console-serving-cert\") pod \"console-fd67b888d-95ckz\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.640340 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.640296 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bf6f910-7936-4266-a167-ebf7c5f2e0af-console-oauth-config\") pod \"console-fd67b888d-95ckz\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.660853 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.660801 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq4bt\" (UniqueName: \"kubernetes.io/projected/2bf6f910-7936-4266-a167-ebf7c5f2e0af-kube-api-access-jq4bt\") pod \"console-fd67b888d-95ckz\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.722332 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.722303 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:27.769018 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.768971 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:46:27.939822 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:27.939725 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:46:28.017366 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:28.017344 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fd67b888d-95ckz"] Oct 30 10:46:28.019868 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:46:28.019840 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf6f910_7936_4266_a167_ebf7c5f2e0af.slice/crio-a817ae7d07b991c5997386e02a32c9ed539b0dd1afff03e81df79d5e43256e71 WatchSource:0}: Error finding container a817ae7d07b991c5997386e02a32c9ed539b0dd1afff03e81df79d5e43256e71: Status 404 returned error can't find the container with id a817ae7d07b991c5997386e02a32c9ed539b0dd1afff03e81df79d5e43256e71 Oct 30 10:46:28.032388 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:28.032277 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd67b888d-95ckz" event={"ID":"2bf6f910-7936-4266-a167-ebf7c5f2e0af","Type":"ContainerStarted","Data":"a817ae7d07b991c5997386e02a32c9ed539b0dd1afff03e81df79d5e43256e71"} Oct 30 10:46:29.037416 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:29.037380 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd67b888d-95ckz" event={"ID":"2bf6f910-7936-4266-a167-ebf7c5f2e0af","Type":"ContainerStarted","Data":"df456b7d24e96af1cffca378ccfdbd9eaecce5313122798ee0f8c8a315d9322b"} Oct 30 10:46:29.092713 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:29.092662 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-fd67b888d-95ckz" podStartSLOduration=2.092648299 podStartE2EDuration="2.092648299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:46:29.080157534 +0000 UTC m=+79.581537019" watchObservedRunningTime="2026-10-30 10:46:29.092648299 +0000 UTC m=+79.594027784" Oct 30 10:46:37.723244 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:37.723208 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:37.723744 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:37.723272 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:37.727780 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:37.727755 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:38.074046 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:38.073953 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:46:38.143222 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:38.143184 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-956f857b5-9m5kc"] Oct 30 10:46:53.471313 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:46:53.471270 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:46:56.257554 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:46:56.257518 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:47:03.165790 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.165734 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-956f857b5-9m5kc" podUID="a4dba19a-6213-4788-a2a4-c42d7f44e76c" containerName="console" containerID="cri-o://268968970af34b17afe16e739bb8bcb6c2a653fed73c5a95d78ddf58e5553293" gracePeriod=15 Oct 30 10:47:03.402018 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.401994 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:47:03.408145 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.408127 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4dba19a-6213-4788-a2a4-c42d7f44e76c-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4dba19a-6213-4788-a2a4-c42d7f44e76c-console-serving-cert\") pod \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " Oct 30 10:47:03.408221 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.408157 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4dba19a-6213-4788-a2a4-c42d7f44e76c-kube-api-access-qlbhj\" (UniqueName: \"kubernetes.io/projected/a4dba19a-6213-4788-a2a4-c42d7f44e76c-kube-api-access-qlbhj\") pod \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " Oct 30 10:47:03.408221 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.408194 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-console-config\" (UniqueName: \"kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-console-config\") pod \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " Oct 30 10:47:03.408321 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.408228 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4dba19a-6213-4788-a2a4-c42d7f44e76c-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4dba19a-6213-4788-a2a4-c42d7f44e76c-console-oauth-config\") pod \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " Oct 30 10:47:03.408321 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.408246 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-service-ca\") pod \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " Oct 30 10:47:03.408321 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.408272 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-trusted-ca-bundle\") pod \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " Oct 30 10:47:03.408321 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.408302 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-oauth-serving-cert\") pod \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " Oct 30 10:47:03.408540 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.408332 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4dba19a-6213-4788-a2a4-c42d7f44e76c-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4dba19a-6213-4788-a2a4-c42d7f44e76c-tmp\") pod \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\" (UID: \"a4dba19a-6213-4788-a2a4-c42d7f44e76c\") " Oct 30 10:47:03.408735 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.408708 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4dba19a-6213-4788-a2a4-c42d7f44e76c-tmp" pod "a4dba19a-6213-4788-a2a4-c42d7f44e76c" (UID: "a4dba19a-6213-4788-a2a4-c42d7f44e76c"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:47:03.408855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.408785 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-service-ca" pod "a4dba19a-6213-4788-a2a4-c42d7f44e76c" (UID: "a4dba19a-6213-4788-a2a4-c42d7f44e76c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:47:03.408855 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.408837 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-oauth-serving-cert" pod "a4dba19a-6213-4788-a2a4-c42d7f44e76c" (UID: "a4dba19a-6213-4788-a2a4-c42d7f44e76c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:47:03.408968 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.408870 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-trusted-ca-bundle" pod "a4dba19a-6213-4788-a2a4-c42d7f44e76c" (UID: "a4dba19a-6213-4788-a2a4-c42d7f44e76c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:47:03.408968 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.408926 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-console-config" pod "a4dba19a-6213-4788-a2a4-c42d7f44e76c" (UID: "a4dba19a-6213-4788-a2a4-c42d7f44e76c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:47:03.410278 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.410258 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4dba19a-6213-4788-a2a4-c42d7f44e76c-console-serving-cert" pod "a4dba19a-6213-4788-a2a4-c42d7f44e76c" (UID: "a4dba19a-6213-4788-a2a4-c42d7f44e76c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:47:03.410799 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.410781 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4dba19a-6213-4788-a2a4-c42d7f44e76c-kube-api-access-qlbhj" pod "a4dba19a-6213-4788-a2a4-c42d7f44e76c" (UID: "a4dba19a-6213-4788-a2a4-c42d7f44e76c"). InnerVolumeSpecName "kube-api-access-qlbhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:47:03.410799 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.410787 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4dba19a-6213-4788-a2a4-c42d7f44e76c-console-oauth-config" pod "a4dba19a-6213-4788-a2a4-c42d7f44e76c" (UID: "a4dba19a-6213-4788-a2a4-c42d7f44e76c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:47:03.509277 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.509247 4289 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4dba19a-6213-4788-a2a4-c42d7f44e76c-console-serving-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:47:03.509277 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.509273 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlbhj\" (UniqueName: \"kubernetes.io/projected/a4dba19a-6213-4788-a2a4-c42d7f44e76c-kube-api-access-qlbhj\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:47:03.509461 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.509284 4289 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-console-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:47:03.509461 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.509292 4289 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a4dba19a-6213-4788-a2a4-c42d7f44e76c-console-oauth-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:47:03.509461 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.509301 4289 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:47:03.509461 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.509309 4289 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-trusted-ca-bundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:47:03.509461 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.509318 4289 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a4dba19a-6213-4788-a2a4-c42d7f44e76c-oauth-serving-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:47:03.509461 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:03.509326 4289 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4dba19a-6213-4788-a2a4-c42d7f44e76c-tmp\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:47:04.162492 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:04.162455 4289 generic.go:356] "Generic (PLEG): container finished" podID="a4dba19a-6213-4788-a2a4-c42d7f44e76c" containerID="268968970af34b17afe16e739bb8bcb6c2a653fed73c5a95d78ddf58e5553293" exitCode=0 Oct 30 10:47:04.162669 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:04.162505 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-956f857b5-9m5kc" Oct 30 10:47:04.162669 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:04.162540 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-956f857b5-9m5kc" event={"ID":"a4dba19a-6213-4788-a2a4-c42d7f44e76c","Type":"ContainerDied","Data":"268968970af34b17afe16e739bb8bcb6c2a653fed73c5a95d78ddf58e5553293"} Oct 30 10:47:04.162669 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:04.162590 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-956f857b5-9m5kc" event={"ID":"a4dba19a-6213-4788-a2a4-c42d7f44e76c","Type":"ContainerDied","Data":"5512ab1e3bf498ab10e458d6ba41ba00135bfc3ed53125fbd474384af5414b3f"} Oct 30 10:47:04.162669 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:04.162615 4289 scope.go:122] "RemoveContainer" containerID="268968970af34b17afe16e739bb8bcb6c2a653fed73c5a95d78ddf58e5553293" Oct 30 10:47:04.171088 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:04.170874 4289 scope.go:122] "RemoveContainer" containerID="268968970af34b17afe16e739bb8bcb6c2a653fed73c5a95d78ddf58e5553293" Oct 30 10:47:04.171347 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:47:04.171172 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"268968970af34b17afe16e739bb8bcb6c2a653fed73c5a95d78ddf58e5553293\": container with ID starting with 268968970af34b17afe16e739bb8bcb6c2a653fed73c5a95d78ddf58e5553293 not found: ID does not exist" containerID="268968970af34b17afe16e739bb8bcb6c2a653fed73c5a95d78ddf58e5553293" Oct 30 10:47:04.171347 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:04.171200 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"268968970af34b17afe16e739bb8bcb6c2a653fed73c5a95d78ddf58e5553293"} err="failed to get container status \"268968970af34b17afe16e739bb8bcb6c2a653fed73c5a95d78ddf58e5553293\": rpc error: code = NotFound desc = could not find container \"268968970af34b17afe16e739bb8bcb6c2a653fed73c5a95d78ddf58e5553293\": container with ID starting with 268968970af34b17afe16e739bb8bcb6c2a653fed73c5a95d78ddf58e5553293 not found: ID does not exist" Oct 30 10:47:04.212352 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:04.212321 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-956f857b5-9m5kc"] Oct 30 10:47:04.224324 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:04.224298 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-956f857b5-9m5kc"] Oct 30 10:47:05.623261 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:05.623224 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4dba19a-6213-4788-a2a4-c42d7f44e76c" path="/var/lib/kubelet/pods/a4dba19a-6213-4788-a2a4-c42d7f44e76c/volumes" Oct 30 10:47:26.257345 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:47:26.257264 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:47:33.870374 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:33.870337 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:47:34.035829 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:34.035793 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:47:36.240176 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.240145 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-747f99888f-6s22s"] Oct 30 10:47:36.240564 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.240410 4289 cpu_manager.go:414] "containerMap: removing container" podUID="a4dba19a-6213-4788-a2a4-c42d7f44e76c" containerName="console" Oct 30 10:47:36.240564 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.240420 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4dba19a-6213-4788-a2a4-c42d7f44e76c" containerName="console" Oct 30 10:47:36.240564 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.240491 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4dba19a-6213-4788-a2a4-c42d7f44e76c" containerName="console" Oct 30 10:47:36.242355 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.242340 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.263940 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.263912 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-747f99888f-6s22s"] Oct 30 10:47:36.345606 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.345575 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/913c87b9-6c0d-43df-80ab-54ab1a9655de-trusted-ca-bundle\") pod \"console-747f99888f-6s22s\" (UID: \"913c87b9-6c0d-43df-80ab-54ab1a9655de\") " pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.345754 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.345622 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/913c87b9-6c0d-43df-80ab-54ab1a9655de-console-serving-cert\") pod \"console-747f99888f-6s22s\" (UID: \"913c87b9-6c0d-43df-80ab-54ab1a9655de\") " pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.345754 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.345646 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/913c87b9-6c0d-43df-80ab-54ab1a9655de-console-oauth-config\") pod \"console-747f99888f-6s22s\" (UID: \"913c87b9-6c0d-43df-80ab-54ab1a9655de\") " pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.345866 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.345755 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/913c87b9-6c0d-43df-80ab-54ab1a9655de-tmp\") pod \"console-747f99888f-6s22s\" (UID: \"913c87b9-6c0d-43df-80ab-54ab1a9655de\") " pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.345866 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.345799 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/913c87b9-6c0d-43df-80ab-54ab1a9655de-service-ca\") pod \"console-747f99888f-6s22s\" (UID: \"913c87b9-6c0d-43df-80ab-54ab1a9655de\") " pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.345949 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.345868 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/913c87b9-6c0d-43df-80ab-54ab1a9655de-console-config\") pod \"console-747f99888f-6s22s\" (UID: \"913c87b9-6c0d-43df-80ab-54ab1a9655de\") " pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.345949 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.345914 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2ldz\" (UniqueName: \"kubernetes.io/projected/913c87b9-6c0d-43df-80ab-54ab1a9655de-kube-api-access-n2ldz\") pod \"console-747f99888f-6s22s\" (UID: \"913c87b9-6c0d-43df-80ab-54ab1a9655de\") " pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.345949 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.345938 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/913c87b9-6c0d-43df-80ab-54ab1a9655de-oauth-serving-cert\") pod \"console-747f99888f-6s22s\" (UID: \"913c87b9-6c0d-43df-80ab-54ab1a9655de\") " pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.432494 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.432465 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:47:36.447038 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.447012 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/913c87b9-6c0d-43df-80ab-54ab1a9655de-oauth-serving-cert\") pod \"console-747f99888f-6s22s\" (UID: \"913c87b9-6c0d-43df-80ab-54ab1a9655de\") " pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.447149 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.447049 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/913c87b9-6c0d-43df-80ab-54ab1a9655de-trusted-ca-bundle\") pod \"console-747f99888f-6s22s\" (UID: \"913c87b9-6c0d-43df-80ab-54ab1a9655de\") " pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.447149 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.447099 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/913c87b9-6c0d-43df-80ab-54ab1a9655de-console-serving-cert\") pod \"console-747f99888f-6s22s\" (UID: \"913c87b9-6c0d-43df-80ab-54ab1a9655de\") " pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.447149 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.447133 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/913c87b9-6c0d-43df-80ab-54ab1a9655de-console-oauth-config\") pod \"console-747f99888f-6s22s\" (UID: \"913c87b9-6c0d-43df-80ab-54ab1a9655de\") " pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.447305 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.447155 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/913c87b9-6c0d-43df-80ab-54ab1a9655de-tmp\") pod \"console-747f99888f-6s22s\" (UID: \"913c87b9-6c0d-43df-80ab-54ab1a9655de\") " pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.447357 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.447304 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/913c87b9-6c0d-43df-80ab-54ab1a9655de-service-ca\") pod \"console-747f99888f-6s22s\" (UID: \"913c87b9-6c0d-43df-80ab-54ab1a9655de\") " pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.447413 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.447364 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/913c87b9-6c0d-43df-80ab-54ab1a9655de-console-config\") pod \"console-747f99888f-6s22s\" (UID: \"913c87b9-6c0d-43df-80ab-54ab1a9655de\") " pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.447413 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.447394 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2ldz\" (UniqueName: \"kubernetes.io/projected/913c87b9-6c0d-43df-80ab-54ab1a9655de-kube-api-access-n2ldz\") pod \"console-747f99888f-6s22s\" (UID: \"913c87b9-6c0d-43df-80ab-54ab1a9655de\") " pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.447925 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.447904 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/913c87b9-6c0d-43df-80ab-54ab1a9655de-oauth-serving-cert\") pod \"console-747f99888f-6s22s\" (UID: \"913c87b9-6c0d-43df-80ab-54ab1a9655de\") " pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.448002 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.447948 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/913c87b9-6c0d-43df-80ab-54ab1a9655de-trusted-ca-bundle\") pod \"console-747f99888f-6s22s\" (UID: \"913c87b9-6c0d-43df-80ab-54ab1a9655de\") " pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.448139 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.448118 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/913c87b9-6c0d-43df-80ab-54ab1a9655de-tmp\") pod \"console-747f99888f-6s22s\" (UID: \"913c87b9-6c0d-43df-80ab-54ab1a9655de\") " pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.448365 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.448347 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/913c87b9-6c0d-43df-80ab-54ab1a9655de-service-ca\") pod \"console-747f99888f-6s22s\" (UID: \"913c87b9-6c0d-43df-80ab-54ab1a9655de\") " pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.448467 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.448410 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/913c87b9-6c0d-43df-80ab-54ab1a9655de-console-config\") pod \"console-747f99888f-6s22s\" (UID: \"913c87b9-6c0d-43df-80ab-54ab1a9655de\") " pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.449608 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.449589 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/913c87b9-6c0d-43df-80ab-54ab1a9655de-console-oauth-config\") pod \"console-747f99888f-6s22s\" (UID: \"913c87b9-6c0d-43df-80ab-54ab1a9655de\") " pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.449748 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.449730 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/913c87b9-6c0d-43df-80ab-54ab1a9655de-console-serving-cert\") pod \"console-747f99888f-6s22s\" (UID: \"913c87b9-6c0d-43df-80ab-54ab1a9655de\") " pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.470277 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.470254 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2ldz\" (UniqueName: \"kubernetes.io/projected/913c87b9-6c0d-43df-80ab-54ab1a9655de-kube-api-access-n2ldz\") pod \"console-747f99888f-6s22s\" (UID: \"913c87b9-6c0d-43df-80ab-54ab1a9655de\") " pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.552403 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.552330 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:36.603669 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.603419 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:47:36.762378 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.762343 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:47:36.846596 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:36.846525 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-747f99888f-6s22s"] Oct 30 10:47:36.849156 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:47:36.849131 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod913c87b9_6c0d_43df_80ab_54ab1a9655de.slice/crio-11c8fb20d10c132c20143f40727e4331009c4a9161498c392507f21e35776803 WatchSource:0}: Error finding container 11c8fb20d10c132c20143f40727e4331009c4a9161498c392507f21e35776803: Status 404 returned error can't find the container with id 11c8fb20d10c132c20143f40727e4331009c4a9161498c392507f21e35776803 Oct 30 10:47:37.273288 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:37.273256 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-747f99888f-6s22s" event={"ID":"913c87b9-6c0d-43df-80ab-54ab1a9655de","Type":"ContainerStarted","Data":"6a55faa9600852632f577d85997e1efe9752765abcf010c4bee11bde7c41ced0"} Oct 30 10:47:37.273678 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:37.273325 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-747f99888f-6s22s" event={"ID":"913c87b9-6c0d-43df-80ab-54ab1a9655de","Type":"ContainerStarted","Data":"11c8fb20d10c132c20143f40727e4331009c4a9161498c392507f21e35776803"} Oct 30 10:47:37.315849 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:37.315808 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-747f99888f-6s22s" podStartSLOduration=1.315797048 podStartE2EDuration="1.315797048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:47:37.315304462 +0000 UTC m=+147.816683983" watchObservedRunningTime="2026-10-30 10:47:37.315797048 +0000 UTC m=+147.817176532" Oct 30 10:47:46.531243 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:46.531208 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:47:46.552966 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:46.552942 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:46.553081 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:46.553068 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:46.557508 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:46.557490 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:46.686350 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:46.686311 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:47:47.311448 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:47.311412 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-747f99888f-6s22s" Oct 30 10:47:47.383592 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:47.383565 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fd67b888d-95ckz"] Oct 30 10:47:53.591264 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:53.591228 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:47:55.894195 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:55.894156 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:47:56.055791 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:47:56.055758 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:47:56.257907 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:47:56.257875 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:48:12.402472 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.402401 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-fd67b888d-95ckz" podUID="2bf6f910-7936-4266-a167-ebf7c5f2e0af" containerName="console" containerID="cri-o://df456b7d24e96af1cffca378ccfdbd9eaecce5313122798ee0f8c8a315d9322b" gracePeriod=15 Oct 30 10:48:12.630348 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.630326 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:48:12.727547 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.727518 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-console-config\" (UniqueName: \"kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-console-config\") pod \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " Oct 30 10:48:12.727675 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.727564 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-oauth-serving-cert\") pod \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " Oct 30 10:48:12.727675 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.727606 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf6f910-7936-4266-a167-ebf7c5f2e0af-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bf6f910-7936-4266-a167-ebf7c5f2e0af-tmp\") pod \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " Oct 30 10:48:12.727675 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.727636 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-service-ca\") pod \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " Oct 30 10:48:12.727675 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.727659 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bf6f910-7936-4266-a167-ebf7c5f2e0af-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bf6f910-7936-4266-a167-ebf7c5f2e0af-console-oauth-config\") pod \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " Oct 30 10:48:12.727906 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.727694 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bf6f910-7936-4266-a167-ebf7c5f2e0af-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bf6f910-7936-4266-a167-ebf7c5f2e0af-console-serving-cert\") pod \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " Oct 30 10:48:12.727906 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.727727 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf6f910-7936-4266-a167-ebf7c5f2e0af-kube-api-access-jq4bt\" (UniqueName: \"kubernetes.io/projected/2bf6f910-7936-4266-a167-ebf7c5f2e0af-kube-api-access-jq4bt\") pod \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " Oct 30 10:48:12.727906 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.727767 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-trusted-ca-bundle\") pod \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\" (UID: \"2bf6f910-7936-4266-a167-ebf7c5f2e0af\") " Oct 30 10:48:12.728054 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.727939 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf6f910-7936-4266-a167-ebf7c5f2e0af-tmp" pod "2bf6f910-7936-4266-a167-ebf7c5f2e0af" (UID: "2bf6f910-7936-4266-a167-ebf7c5f2e0af"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 10:48:12.728109 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.728062 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-console-config" pod "2bf6f910-7936-4266-a167-ebf7c5f2e0af" (UID: "2bf6f910-7936-4266-a167-ebf7c5f2e0af"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:48:12.728109 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.728096 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-oauth-serving-cert" pod "2bf6f910-7936-4266-a167-ebf7c5f2e0af" (UID: "2bf6f910-7936-4266-a167-ebf7c5f2e0af"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:48:12.728276 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.728241 4289 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bf6f910-7936-4266-a167-ebf7c5f2e0af-tmp\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:48:12.728412 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.728378 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-trusted-ca-bundle" pod "2bf6f910-7936-4266-a167-ebf7c5f2e0af" (UID: "2bf6f910-7936-4266-a167-ebf7c5f2e0af"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:48:12.728412 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.728375 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-service-ca" pod "2bf6f910-7936-4266-a167-ebf7c5f2e0af" (UID: "2bf6f910-7936-4266-a167-ebf7c5f2e0af"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:48:12.730132 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.730111 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf6f910-7936-4266-a167-ebf7c5f2e0af-kube-api-access-jq4bt" pod "2bf6f910-7936-4266-a167-ebf7c5f2e0af" (UID: "2bf6f910-7936-4266-a167-ebf7c5f2e0af"). InnerVolumeSpecName "kube-api-access-jq4bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:48:12.730410 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.730393 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bf6f910-7936-4266-a167-ebf7c5f2e0af-console-oauth-config" pod "2bf6f910-7936-4266-a167-ebf7c5f2e0af" (UID: "2bf6f910-7936-4266-a167-ebf7c5f2e0af"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:48:12.730519 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.730422 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bf6f910-7936-4266-a167-ebf7c5f2e0af-console-serving-cert" pod "2bf6f910-7936-4266-a167-ebf7c5f2e0af" (UID: "2bf6f910-7936-4266-a167-ebf7c5f2e0af"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:48:12.828786 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.828752 4289 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bf6f910-7936-4266-a167-ebf7c5f2e0af-console-serving-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:48:12.828786 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.828779 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq4bt\" (UniqueName: \"kubernetes.io/projected/2bf6f910-7936-4266-a167-ebf7c5f2e0af-kube-api-access-jq4bt\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:48:12.828786 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.828789 4289 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-trusted-ca-bundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:48:12.829016 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.828799 4289 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-console-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:48:12.829016 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.828808 4289 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-oauth-serving-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:48:12.829016 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.828817 4289 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf6f910-7936-4266-a167-ebf7c5f2e0af-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:48:12.829016 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:12.828826 4289 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bf6f910-7936-4266-a167-ebf7c5f2e0af-console-oauth-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:48:13.400180 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:13.400146 4289 generic.go:356] "Generic (PLEG): container finished" podID="2bf6f910-7936-4266-a167-ebf7c5f2e0af" containerID="df456b7d24e96af1cffca378ccfdbd9eaecce5313122798ee0f8c8a315d9322b" exitCode=0 Oct 30 10:48:13.400355 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:13.400195 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd67b888d-95ckz" Oct 30 10:48:13.400355 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:13.400215 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd67b888d-95ckz" event={"ID":"2bf6f910-7936-4266-a167-ebf7c5f2e0af","Type":"ContainerDied","Data":"df456b7d24e96af1cffca378ccfdbd9eaecce5313122798ee0f8c8a315d9322b"} Oct 30 10:48:13.400355 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:13.400252 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd67b888d-95ckz" event={"ID":"2bf6f910-7936-4266-a167-ebf7c5f2e0af","Type":"ContainerDied","Data":"a817ae7d07b991c5997386e02a32c9ed539b0dd1afff03e81df79d5e43256e71"} Oct 30 10:48:13.400355 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:13.400271 4289 scope.go:122] "RemoveContainer" containerID="df456b7d24e96af1cffca378ccfdbd9eaecce5313122798ee0f8c8a315d9322b" Oct 30 10:48:13.408740 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:13.408571 4289 scope.go:122] "RemoveContainer" containerID="df456b7d24e96af1cffca378ccfdbd9eaecce5313122798ee0f8c8a315d9322b" Oct 30 10:48:13.408992 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:48:13.408849 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df456b7d24e96af1cffca378ccfdbd9eaecce5313122798ee0f8c8a315d9322b\": container with ID starting with df456b7d24e96af1cffca378ccfdbd9eaecce5313122798ee0f8c8a315d9322b not found: ID does not exist" containerID="df456b7d24e96af1cffca378ccfdbd9eaecce5313122798ee0f8c8a315d9322b" Oct 30 10:48:13.408992 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:13.408882 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df456b7d24e96af1cffca378ccfdbd9eaecce5313122798ee0f8c8a315d9322b"} err="failed to get container status \"df456b7d24e96af1cffca378ccfdbd9eaecce5313122798ee0f8c8a315d9322b\": rpc error: code = NotFound desc = could not find container \"df456b7d24e96af1cffca378ccfdbd9eaecce5313122798ee0f8c8a315d9322b\": container with ID starting with df456b7d24e96af1cffca378ccfdbd9eaecce5313122798ee0f8c8a315d9322b not found: ID does not exist" Oct 30 10:48:13.449234 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:13.449212 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fd67b888d-95ckz"] Oct 30 10:48:13.461001 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:13.460977 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fd67b888d-95ckz"] Oct 30 10:48:13.623852 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:13.623820 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bf6f910-7936-4266-a167-ebf7c5f2e0af" path="/var/lib/kubelet/pods/2bf6f910-7936-4266-a167-ebf7c5f2e0af/volumes" Oct 30 10:48:26.257243 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:48:26.257207 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:48:53.726113 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:48:53.726032 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:48:56.257389 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:48:56.257344 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:49:26.256558 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:49:26.256522 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:49:53.839197 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:49:53.839156 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:49:56.256840 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:49:56.256795 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:50:09.535859 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:09.535819 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:50:09.722016 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:09.721976 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:50:15.592484 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:15.592444 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:50:26.256800 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:50:26.256726 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:50:26.367193 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:26.367151 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:50:26.530882 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:26.530798 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:50:29.300881 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:29.300842 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:50:31.636873 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:31.636839 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:50:33.035010 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:33.034978 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:50:34.849985 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:34.849945 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:50:35.021591 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:35.021552 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:50:39.397900 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:39.397843 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:50:39.593576 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:39.593546 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:50:40.252388 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:40.252344 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:50:40.424031 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:40.424001 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:50:41.449517 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:41.449477 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:50:41.820648 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:41.820617 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:50:41.974551 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:41.974519 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-hpkqj"] Oct 30 10:50:41.974800 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:41.974789 4289 cpu_manager.go:414] "containerMap: removing container" podUID="2bf6f910-7936-4266-a167-ebf7c5f2e0af" containerName="console" Oct 30 10:50:41.974842 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:41.974801 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf6f910-7936-4266-a167-ebf7c5f2e0af" containerName="console" Oct 30 10:50:41.974876 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:41.974859 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf6f910-7936-4266-a167-ebf7c5f2e0af" containerName="console" Oct 30 10:50:41.976759 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:41.976739 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hpkqj" Oct 30 10:50:41.989081 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:41.989061 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 10:50:41.989207 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:41.989159 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 10:50:41.999265 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:41.999244 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-gptn6\"" Oct 30 10:50:42.000341 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:42.000318 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-hpkqj"] Oct 30 10:50:42.114460 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:42.114344 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a22cd35d-59cb-422a-951a-5a60928856e5-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-hpkqj\" (UID: \"a22cd35d-59cb-422a-951a-5a60928856e5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hpkqj" Oct 30 10:50:42.114460 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:42.114395 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcwmb\" (UniqueName: \"kubernetes.io/projected/a22cd35d-59cb-422a-951a-5a60928856e5-kube-api-access-fcwmb\") pod \"cert-manager-cainjector-5c57bbd4fc-hpkqj\" (UID: \"a22cd35d-59cb-422a-951a-5a60928856e5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hpkqj" Oct 30 10:50:42.215247 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:42.215214 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a22cd35d-59cb-422a-951a-5a60928856e5-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-hpkqj\" (UID: \"a22cd35d-59cb-422a-951a-5a60928856e5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hpkqj" Oct 30 10:50:42.215480 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:42.215276 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcwmb\" (UniqueName: \"kubernetes.io/projected/a22cd35d-59cb-422a-951a-5a60928856e5-kube-api-access-fcwmb\") pod \"cert-manager-cainjector-5c57bbd4fc-hpkqj\" (UID: \"a22cd35d-59cb-422a-951a-5a60928856e5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hpkqj" Oct 30 10:50:42.236148 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:42.236126 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a22cd35d-59cb-422a-951a-5a60928856e5-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-hpkqj\" (UID: \"a22cd35d-59cb-422a-951a-5a60928856e5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hpkqj" Oct 30 10:50:42.236353 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:42.236333 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcwmb\" (UniqueName: \"kubernetes.io/projected/a22cd35d-59cb-422a-951a-5a60928856e5-kube-api-access-fcwmb\") pod \"cert-manager-cainjector-5c57bbd4fc-hpkqj\" (UID: \"a22cd35d-59cb-422a-951a-5a60928856e5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hpkqj" Oct 30 10:50:42.303124 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:42.303091 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hpkqj" Oct 30 10:50:42.590713 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:42.590680 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-hpkqj"] Oct 30 10:50:42.599662 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:42.599642 4289 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 10:50:42.898492 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:42.898408 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hpkqj" event={"ID":"a22cd35d-59cb-422a-951a-5a60928856e5","Type":"ContainerStarted","Data":"94e1cc395ac319c532f7d5f33bc27ed9a863dd057911a30b23aa6a09ab21607a"} Oct 30 10:50:45.912473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:45.912366 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hpkqj" event={"ID":"a22cd35d-59cb-422a-951a-5a60928856e5","Type":"ContainerStarted","Data":"3ed2afb1ded7967e50fe0a4279b03ff21ffbabfebb33ac8a799c10f5ef9e45df"} Oct 30 10:50:45.949139 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:45.949087 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hpkqj" podStartSLOduration=2.023027916 podStartE2EDuration="4.9490741s" totalImagesPullingTime="2.926046184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:50:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:50:45.948123602 +0000 UTC m=+336.449503082" watchObservedRunningTime="2026-10-30 10:50:45.9490741 +0000 UTC m=+336.450453584" Oct 30 10:50:49.657661 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:49.657624 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:50:49.907932 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:49.907840 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:50:50.087675 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:50.087637 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:50:50.263994 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:50.263955 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:50:53.940477 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:50:53.940443 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:50:56.256642 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:50:56.256607 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:51:01.060716 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:01.060676 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:51:01.156335 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:01.156306 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-observability-system/odh-observability-54787bff44-25dxm"] Oct 30 10:51:01.159502 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:01.159482 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-observability-system/odh-observability-54787bff44-25dxm" Oct 30 10:51:01.174633 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:01.174603 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-observability-system\"/\"odh-observability-webhook-cert\"" Oct 30 10:51:01.188473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:01.188022 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-observability-system\"/\"odh-observability-dockercfg-4xvjm\"" Oct 30 10:51:01.188473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:01.188052 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-observability-system\"/\"odh-observability-metrics-tls\"" Oct 30 10:51:01.188473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:01.188052 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-observability-system\"/\"openshift-service-ca.crt\"" Oct 30 10:51:01.188473 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:01.188109 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-observability-system\"/\"kube-root-ca.crt\"" Oct 30 10:51:01.189377 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:01.189353 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-observability-system/odh-observability-54787bff44-25dxm"] Oct 30 10:51:01.240881 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:01.240850 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:51:01.262533 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:01.262508 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17e85285-af17-408d-802b-47ae882f8a90-metrics-tls\") pod \"odh-observability-54787bff44-25dxm\" (UID: \"17e85285-af17-408d-802b-47ae882f8a90\") " pod="odh-observability-system/odh-observability-54787bff44-25dxm" Oct 30 10:51:01.262676 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:01.262551 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/17e85285-af17-408d-802b-47ae882f8a90-webhook-certs\") pod \"odh-observability-54787bff44-25dxm\" (UID: \"17e85285-af17-408d-802b-47ae882f8a90\") " pod="odh-observability-system/odh-observability-54787bff44-25dxm" Oct 30 10:51:01.262676 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:01.262584 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j76n\" (UniqueName: \"kubernetes.io/projected/17e85285-af17-408d-802b-47ae882f8a90-kube-api-access-9j76n\") pod \"odh-observability-54787bff44-25dxm\" (UID: \"17e85285-af17-408d-802b-47ae882f8a90\") " pod="odh-observability-system/odh-observability-54787bff44-25dxm" Oct 30 10:51:01.363862 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:01.363787 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j76n\" (UniqueName: \"kubernetes.io/projected/17e85285-af17-408d-802b-47ae882f8a90-kube-api-access-9j76n\") pod \"odh-observability-54787bff44-25dxm\" (UID: \"17e85285-af17-408d-802b-47ae882f8a90\") " pod="odh-observability-system/odh-observability-54787bff44-25dxm" Oct 30 10:51:01.363862 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:01.363836 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17e85285-af17-408d-802b-47ae882f8a90-metrics-tls\") pod \"odh-observability-54787bff44-25dxm\" (UID: \"17e85285-af17-408d-802b-47ae882f8a90\") " pod="odh-observability-system/odh-observability-54787bff44-25dxm" Oct 30 10:51:01.364077 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:01.363867 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/17e85285-af17-408d-802b-47ae882f8a90-webhook-certs\") pod \"odh-observability-54787bff44-25dxm\" (UID: \"17e85285-af17-408d-802b-47ae882f8a90\") " pod="odh-observability-system/odh-observability-54787bff44-25dxm" Oct 30 10:51:01.364077 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:51:01.363964 4289 secret.go:189] Couldn't get secret odh-observability-system/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 30 10:51:01.364077 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:51:01.364029 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17e85285-af17-408d-802b-47ae882f8a90-webhook-certs podName:17e85285-af17-408d-802b-47ae882f8a90 nodeName:}" failed. No retries permitted until 2026-10-30 10:51:01.864009453 +0000 UTC m=+352.365388934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/17e85285-af17-408d-802b-47ae882f8a90-webhook-certs") pod "odh-observability-54787bff44-25dxm" (UID: "17e85285-af17-408d-802b-47ae882f8a90") : secret "odh-observability-webhook-cert" not found Oct 30 10:51:01.366323 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:01.366297 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17e85285-af17-408d-802b-47ae882f8a90-metrics-tls\") pod \"odh-observability-54787bff44-25dxm\" (UID: \"17e85285-af17-408d-802b-47ae882f8a90\") " pod="odh-observability-system/odh-observability-54787bff44-25dxm" Oct 30 10:51:01.387129 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:01.387108 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j76n\" (UniqueName: \"kubernetes.io/projected/17e85285-af17-408d-802b-47ae882f8a90-kube-api-access-9j76n\") pod \"odh-observability-54787bff44-25dxm\" (UID: \"17e85285-af17-408d-802b-47ae882f8a90\") " pod="odh-observability-system/odh-observability-54787bff44-25dxm" Oct 30 10:51:01.409991 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:01.409963 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:51:01.868485 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:01.868441 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/17e85285-af17-408d-802b-47ae882f8a90-webhook-certs\") pod \"odh-observability-54787bff44-25dxm\" (UID: \"17e85285-af17-408d-802b-47ae882f8a90\") " pod="odh-observability-system/odh-observability-54787bff44-25dxm" Oct 30 10:51:01.868664 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:51:01.868559 4289 secret.go:189] Couldn't get secret odh-observability-system/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 30 10:51:01.868664 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:51:01.868617 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17e85285-af17-408d-802b-47ae882f8a90-webhook-certs podName:17e85285-af17-408d-802b-47ae882f8a90 nodeName:}" failed. No retries permitted until 2026-10-30 10:51:02.868603427 +0000 UTC m=+353.369982911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/17e85285-af17-408d-802b-47ae882f8a90-webhook-certs") pod "odh-observability-54787bff44-25dxm" (UID: "17e85285-af17-408d-802b-47ae882f8a90") : secret "odh-observability-webhook-cert" not found Oct 30 10:51:02.874218 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:02.874177 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/17e85285-af17-408d-802b-47ae882f8a90-webhook-certs\") pod \"odh-observability-54787bff44-25dxm\" (UID: \"17e85285-af17-408d-802b-47ae882f8a90\") " pod="odh-observability-system/odh-observability-54787bff44-25dxm" Oct 30 10:51:02.874685 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:51:02.874309 4289 secret.go:189] Couldn't get secret odh-observability-system/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 30 10:51:02.874685 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:51:02.874366 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17e85285-af17-408d-802b-47ae882f8a90-webhook-certs podName:17e85285-af17-408d-802b-47ae882f8a90 nodeName:}" failed. No retries permitted until 2026-10-30 10:51:04.874353234 +0000 UTC m=+355.375732697 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/17e85285-af17-408d-802b-47ae882f8a90-webhook-certs") pod "odh-observability-54787bff44-25dxm" (UID: "17e85285-af17-408d-802b-47ae882f8a90") : secret "odh-observability-webhook-cert" not found Oct 30 10:51:04.890983 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:04.890938 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/17e85285-af17-408d-802b-47ae882f8a90-webhook-certs\") pod \"odh-observability-54787bff44-25dxm\" (UID: \"17e85285-af17-408d-802b-47ae882f8a90\") " pod="odh-observability-system/odh-observability-54787bff44-25dxm" Oct 30 10:51:04.891396 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:51:04.891081 4289 secret.go:189] Couldn't get secret odh-observability-system/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 30 10:51:04.891396 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:51:04.891155 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17e85285-af17-408d-802b-47ae882f8a90-webhook-certs podName:17e85285-af17-408d-802b-47ae882f8a90 nodeName:}" failed. No retries permitted until 2026-10-30 10:51:08.891135861 +0000 UTC m=+359.392515325 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/17e85285-af17-408d-802b-47ae882f8a90-webhook-certs") pod "odh-observability-54787bff44-25dxm" (UID: "17e85285-af17-408d-802b-47ae882f8a90") : secret "odh-observability-webhook-cert" not found Oct 30 10:51:08.921772 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:08.921724 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/17e85285-af17-408d-802b-47ae882f8a90-webhook-certs\") pod \"odh-observability-54787bff44-25dxm\" (UID: \"17e85285-af17-408d-802b-47ae882f8a90\") " pod="odh-observability-system/odh-observability-54787bff44-25dxm" Oct 30 10:51:08.922251 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:51:08.921842 4289 secret.go:189] Couldn't get secret odh-observability-system/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 30 10:51:08.922251 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:51:08.921902 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17e85285-af17-408d-802b-47ae882f8a90-webhook-certs podName:17e85285-af17-408d-802b-47ae882f8a90 nodeName:}" failed. No retries permitted until 2026-10-30 10:51:16.921889042 +0000 UTC m=+367.423268506 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/17e85285-af17-408d-802b-47ae882f8a90-webhook-certs") pod "odh-observability-54787bff44-25dxm" (UID: "17e85285-af17-408d-802b-47ae882f8a90") : secret "odh-observability-webhook-cert" not found Oct 30 10:51:16.983496 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:16.983459 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/17e85285-af17-408d-802b-47ae882f8a90-webhook-certs\") pod \"odh-observability-54787bff44-25dxm\" (UID: \"17e85285-af17-408d-802b-47ae882f8a90\") " pod="odh-observability-system/odh-observability-54787bff44-25dxm" Oct 30 10:51:16.983927 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:51:16.983576 4289 secret.go:189] Couldn't get secret odh-observability-system/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 30 10:51:16.983927 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:51:16.983636 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17e85285-af17-408d-802b-47ae882f8a90-webhook-certs podName:17e85285-af17-408d-802b-47ae882f8a90 nodeName:}" failed. No retries permitted until 2026-10-30 10:51:32.983622872 +0000 UTC m=+383.485002334 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/17e85285-af17-408d-802b-47ae882f8a90-webhook-certs") pod "odh-observability-54787bff44-25dxm" (UID: "17e85285-af17-408d-802b-47ae882f8a90") : secret "odh-observability-webhook-cert" not found Oct 30 10:51:21.298781 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:21.298738 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:51:26.256706 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:51:26.256668 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:51:33.014947 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:33.014908 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/17e85285-af17-408d-802b-47ae882f8a90-webhook-certs\") pod \"odh-observability-54787bff44-25dxm\" (UID: \"17e85285-af17-408d-802b-47ae882f8a90\") " pod="odh-observability-system/odh-observability-54787bff44-25dxm" Oct 30 10:51:33.017296 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:33.017267 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/17e85285-af17-408d-802b-47ae882f8a90-webhook-certs\") pod \"odh-observability-54787bff44-25dxm\" (UID: \"17e85285-af17-408d-802b-47ae882f8a90\") " pod="odh-observability-system/odh-observability-54787bff44-25dxm" Oct 30 10:51:33.283046 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:33.282957 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-observability-system\"/\"odh-observability-dockercfg-4xvjm\"" Oct 30 10:51:33.292017 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:33.291991 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-observability-system/odh-observability-54787bff44-25dxm" Oct 30 10:51:33.584323 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:33.584248 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-observability-system/odh-observability-54787bff44-25dxm"] Oct 30 10:51:33.588954 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:51:33.588928 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e85285_af17_408d_802b_47ae882f8a90.slice/crio-2a8126f9441f4749ab8ccc46d0eb7498a4b6131a5d490db1fa1b42c7975137e5 WatchSource:0}: Error finding container 2a8126f9441f4749ab8ccc46d0eb7498a4b6131a5d490db1fa1b42c7975137e5: Status 404 returned error can't find the container with id 2a8126f9441f4749ab8ccc46d0eb7498a4b6131a5d490db1fa1b42c7975137e5 Oct 30 10:51:34.083660 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:34.083626 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-observability-system/odh-observability-54787bff44-25dxm" event={"ID":"17e85285-af17-408d-802b-47ae882f8a90","Type":"ContainerStarted","Data":"2a8126f9441f4749ab8ccc46d0eb7498a4b6131a5d490db1fa1b42c7975137e5"} Oct 30 10:51:36.093533 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:36.093411 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-observability-system/odh-observability-54787bff44-25dxm" event={"ID":"17e85285-af17-408d-802b-47ae882f8a90","Type":"ContainerStarted","Data":"04ae1983cbe015fbadc003a188f9f09fab00e601e7905e7300ef3105dd4b10e2"} Oct 30 10:51:36.094018 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:36.093599 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-observability-system/odh-observability-54787bff44-25dxm" Oct 30 10:51:36.131362 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:36.131311 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-observability-system/odh-observability-54787bff44-25dxm" podStartSLOduration=32.912963732 podStartE2EDuration="35.131298498s" totalImagesPullingTime="2.218334766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:51:36.130771194 +0000 UTC m=+386.632150681" watchObservedRunningTime="2026-10-30 10:51:36.131298498 +0000 UTC m=+386.632677982" Oct 30 10:51:47.099381 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:47.099351 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-observability-system/odh-observability-54787bff44-25dxm" Oct 30 10:51:54.070065 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:51:54.069965 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:51:56.257094 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:51:56.257051 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:52:26.256770 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:52:26.256733 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:52:41.519570 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:52:41.519526 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:52:41.695498 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:52:41.695461 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:52:51.486182 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:52:51.486144 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:52:51.653834 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:52:51.653800 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:52:54.190590 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:52:54.190557 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:52:56.257173 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:52:56.257138 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:53:00.015517 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:00.015476 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:00.187261 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:00.187230 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:00.498834 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:00.498803 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:00.668897 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:00.668858 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:01.922004 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:01.921970 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:03.018869 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:03.018828 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:03.190467 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:03.190405 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:03.673242 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:03.673195 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:03.853502 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:03.853466 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:04.028318 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:04.028286 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:04.699160 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:04.699125 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:04.863899 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:04.863873 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:05.790834 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:05.790798 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:06.122993 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:06.122907 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:06.286629 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:06.286592 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:06.454113 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:06.454025 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:07.883633 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:07.883597 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:08.082201 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:08.082154 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:08.304305 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:08.304267 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:08.625980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:08.625890 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:09.118338 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:09.118302 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:09.893621 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:09.893587 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:10.512387 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:10.512350 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:10.621105 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:10.621070 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-bd4f6797b-pjlxd"] Oct 30 10:53:10.624412 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:10.624382 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-bd4f6797b-pjlxd" Oct 30 10:53:10.638274 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:10.638249 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-opentelemetry-operator\"/\"openshift-service-ca.crt\"" Oct 30 10:53:10.638385 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:10.638307 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-opentelemetry-operator\"/\"kube-root-ca.crt\"" Oct 30 10:53:10.647290 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:10.647274 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-controller-manager-dockercfg-tdlw7\"" Oct 30 10:53:10.647342 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:10.647296 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-webhook-service-cert\"" Oct 30 10:53:10.651770 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:10.651750 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-bd4f6797b-pjlxd"] Oct 30 10:53:10.700024 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:10.699984 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:10.705992 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:10.705956 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f6b8dbeb-9db2-4122-bae2-cdafa9f1af2e-apiservice-cert\") pod \"opentelemetry-operator-webhook-bd4f6797b-pjlxd\" (UID: \"f6b8dbeb-9db2-4122-bae2-cdafa9f1af2e\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-bd4f6797b-pjlxd" Oct 30 10:53:10.706064 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:10.706002 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6b8dbeb-9db2-4122-bae2-cdafa9f1af2e-webhook-cert\") pod \"opentelemetry-operator-webhook-bd4f6797b-pjlxd\" (UID: \"f6b8dbeb-9db2-4122-bae2-cdafa9f1af2e\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-bd4f6797b-pjlxd" Oct 30 10:53:10.706111 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:10.706089 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xdvl\" (UniqueName: \"kubernetes.io/projected/f6b8dbeb-9db2-4122-bae2-cdafa9f1af2e-kube-api-access-6xdvl\") pod \"opentelemetry-operator-webhook-bd4f6797b-pjlxd\" (UID: \"f6b8dbeb-9db2-4122-bae2-cdafa9f1af2e\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-bd4f6797b-pjlxd" Oct 30 10:53:10.806708 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:10.806621 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xdvl\" (UniqueName: \"kubernetes.io/projected/f6b8dbeb-9db2-4122-bae2-cdafa9f1af2e-kube-api-access-6xdvl\") pod \"opentelemetry-operator-webhook-bd4f6797b-pjlxd\" (UID: \"f6b8dbeb-9db2-4122-bae2-cdafa9f1af2e\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-bd4f6797b-pjlxd" Oct 30 10:53:10.806708 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:10.806681 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f6b8dbeb-9db2-4122-bae2-cdafa9f1af2e-apiservice-cert\") pod \"opentelemetry-operator-webhook-bd4f6797b-pjlxd\" (UID: \"f6b8dbeb-9db2-4122-bae2-cdafa9f1af2e\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-bd4f6797b-pjlxd" Oct 30 10:53:10.806937 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:10.806710 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6b8dbeb-9db2-4122-bae2-cdafa9f1af2e-webhook-cert\") pod \"opentelemetry-operator-webhook-bd4f6797b-pjlxd\" (UID: \"f6b8dbeb-9db2-4122-bae2-cdafa9f1af2e\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-bd4f6797b-pjlxd" Oct 30 10:53:10.809176 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:10.809148 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f6b8dbeb-9db2-4122-bae2-cdafa9f1af2e-apiservice-cert\") pod \"opentelemetry-operator-webhook-bd4f6797b-pjlxd\" (UID: \"f6b8dbeb-9db2-4122-bae2-cdafa9f1af2e\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-bd4f6797b-pjlxd" Oct 30 10:53:10.809176 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:10.809167 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6b8dbeb-9db2-4122-bae2-cdafa9f1af2e-webhook-cert\") pod \"opentelemetry-operator-webhook-bd4f6797b-pjlxd\" (UID: \"f6b8dbeb-9db2-4122-bae2-cdafa9f1af2e\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-bd4f6797b-pjlxd" Oct 30 10:53:10.835068 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:10.835045 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xdvl\" (UniqueName: \"kubernetes.io/projected/f6b8dbeb-9db2-4122-bae2-cdafa9f1af2e-kube-api-access-6xdvl\") pod \"opentelemetry-operator-webhook-bd4f6797b-pjlxd\" (UID: \"f6b8dbeb-9db2-4122-bae2-cdafa9f1af2e\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-bd4f6797b-pjlxd" Oct 30 10:53:10.935779 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:10.935740 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-bd4f6797b-pjlxd" Oct 30 10:53:11.254931 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:11.254898 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-bd4f6797b-pjlxd"] Oct 30 10:53:11.259877 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:53:11.259842 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b8dbeb_9db2_4122_bae2_cdafa9f1af2e.slice/crio-b19dd5544860b254d17a376b6dcc6324e6fa661a96214b77e22e08aa246f0481 WatchSource:0}: Error finding container b19dd5544860b254d17a376b6dcc6324e6fa661a96214b77e22e08aa246f0481: Status 404 returned error can't find the container with id b19dd5544860b254d17a376b6dcc6324e6fa661a96214b77e22e08aa246f0481 Oct 30 10:53:11.426979 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:11.426937 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-bd4f6797b-pjlxd" event={"ID":"f6b8dbeb-9db2-4122-bae2-cdafa9f1af2e","Type":"ContainerStarted","Data":"b19dd5544860b254d17a376b6dcc6324e6fa661a96214b77e22e08aa246f0481"} Oct 30 10:53:12.225213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:12.225180 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:12.441579 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:12.441542 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:13.436544 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:13.436506 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-bd4f6797b-pjlxd" event={"ID":"f6b8dbeb-9db2-4122-bae2-cdafa9f1af2e","Type":"ContainerStarted","Data":"6a4bd001b8e3456cd820498e2d8ee7244a2635746a5c9a4c23bfb33d2417719a"} Oct 30 10:53:13.436953 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:13.436602 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-bd4f6797b-pjlxd" Oct 30 10:53:13.537851 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:13.537799 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-bd4f6797b-pjlxd" podStartSLOduration=2.177886761 podStartE2EDuration="3.537785385s" totalImagesPullingTime="1.359898624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:53:13.526973725 +0000 UTC m=+484.028353210" watchObservedRunningTime="2026-10-30 10:53:13.537785385 +0000 UTC m=+484.039164867" Oct 30 10:53:14.136497 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:14.136464 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:14.436119 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:14.436032 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:14.905400 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:14.905360 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:15.075748 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:15.075714 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:15.586179 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:15.586138 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:15.746205 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:15.746169 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:17.207190 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:17.207157 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:17.388064 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:17.388030 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:17.820712 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:17.820672 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:18.094379 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:18.094298 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:18.586481 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:18.586447 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:18.776448 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:18.776399 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:20.827025 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:20.826990 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:21.020502 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.020467 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:21.040030 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.040004 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-9wwp4"] Oct 30 10:53:21.043323 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.043309 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-9wwp4" Oct 30 10:53:21.060849 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.060826 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-observability-operator\"/\"kube-root-ca.crt\"" Oct 30 10:53:21.067900 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.067876 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-observability-operator\"/\"openshift-service-ca.crt\"" Oct 30 10:53:21.067900 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.067893 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"obo-prometheus-operator-dockercfg-95smm\"" Oct 30 10:53:21.069606 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.069585 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-9wwp4"] Oct 30 10:53:21.091066 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.091000 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8msl\" (UniqueName: \"kubernetes.io/projected/61a6f0bb-fdca-4a0c-84b5-884a8dea32f8-kube-api-access-n8msl\") pod \"obo-prometheus-operator-6ddb664c4f-9wwp4\" (UID: \"61a6f0bb-fdca-4a0c-84b5-884a8dea32f8\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-9wwp4" Oct 30 10:53:21.191558 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.191530 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8msl\" (UniqueName: \"kubernetes.io/projected/61a6f0bb-fdca-4a0c-84b5-884a8dea32f8-kube-api-access-n8msl\") pod \"obo-prometheus-operator-6ddb664c4f-9wwp4\" (UID: \"61a6f0bb-fdca-4a0c-84b5-884a8dea32f8\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-9wwp4" Oct 30 10:53:21.193515 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.193489 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:21.213205 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.213183 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8msl\" (UniqueName: \"kubernetes.io/projected/61a6f0bb-fdca-4a0c-84b5-884a8dea32f8-kube-api-access-n8msl\") pod \"obo-prometheus-operator-6ddb664c4f-9wwp4\" (UID: \"61a6f0bb-fdca-4a0c-84b5-884a8dea32f8\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-9wwp4" Oct 30 10:53:21.354968 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.354890 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-9wwp4" Oct 30 10:53:21.393336 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.393306 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:21.623816 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.623750 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-748bf975df-rlcvl"] Oct 30 10:53:21.627095 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.627079 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-748bf975df-rlcvl" Oct 30 10:53:21.651307 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.651251 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"obo-prometheus-operator-admission-webhook-service-cert\"" Oct 30 10:53:21.651452 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.651324 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"obo-prometheus-operator-admission-webhook-dockercfg-9hbqg\"" Oct 30 10:53:21.662652 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.662629 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-748bf975df-rlcvl"] Oct 30 10:53:21.674801 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.674779 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-9wwp4"] Oct 30 10:53:21.677205 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.677182 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:21.681066 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:53:21.681043 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a6f0bb_fdca_4a0c_84b5_884a8dea32f8.slice/crio-ca4063227ac6580146afd27eecb9faa61583498f10c582ad2513a739957bdebc WatchSource:0}: Error finding container ca4063227ac6580146afd27eecb9faa61583498f10c582ad2513a739957bdebc: Status 404 returned error can't find the container with id ca4063227ac6580146afd27eecb9faa61583498f10c582ad2513a739957bdebc Oct 30 10:53:21.695108 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.695084 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/decaf67e-662e-4f61-82ea-3f1f1ed143ff-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-748bf975df-rlcvl\" (UID: \"decaf67e-662e-4f61-82ea-3f1f1ed143ff\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-748bf975df-rlcvl" Oct 30 10:53:21.695203 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.695155 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/decaf67e-662e-4f61-82ea-3f1f1ed143ff-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-748bf975df-rlcvl\" (UID: \"decaf67e-662e-4f61-82ea-3f1f1ed143ff\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-748bf975df-rlcvl" Oct 30 10:53:21.796119 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.796084 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/decaf67e-662e-4f61-82ea-3f1f1ed143ff-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-748bf975df-rlcvl\" (UID: \"decaf67e-662e-4f61-82ea-3f1f1ed143ff\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-748bf975df-rlcvl" Oct 30 10:53:21.796271 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.796143 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/decaf67e-662e-4f61-82ea-3f1f1ed143ff-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-748bf975df-rlcvl\" (UID: \"decaf67e-662e-4f61-82ea-3f1f1ed143ff\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-748bf975df-rlcvl" Oct 30 10:53:21.798499 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.798470 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/decaf67e-662e-4f61-82ea-3f1f1ed143ff-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-748bf975df-rlcvl\" (UID: \"decaf67e-662e-4f61-82ea-3f1f1ed143ff\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-748bf975df-rlcvl" Oct 30 10:53:21.798499 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.798494 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/decaf67e-662e-4f61-82ea-3f1f1ed143ff-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-748bf975df-rlcvl\" (UID: \"decaf67e-662e-4f61-82ea-3f1f1ed143ff\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-748bf975df-rlcvl" Oct 30 10:53:21.871072 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.871036 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:21.939562 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:21.939485 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-748bf975df-rlcvl" Oct 30 10:53:22.105883 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:22.105846 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-observability-operator/observability-operator-599c6f89d-5kvt2"] Oct 30 10:53:22.109383 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:22.109365 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-5kvt2" Oct 30 10:53:22.123664 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:22.123643 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"observability-operator-sa-dockercfg-khb6f\"" Oct 30 10:53:22.123804 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:22.123791 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"observability-operator-tls\"" Oct 30 10:53:22.136592 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:22.136566 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/observability-operator-599c6f89d-5kvt2"] Oct 30 10:53:22.199690 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:22.199628 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mktm\" (UniqueName: \"kubernetes.io/projected/3cc54d0f-c711-4bac-96f8-5254b87278b6-kube-api-access-8mktm\") pod \"observability-operator-599c6f89d-5kvt2\" (UID: \"3cc54d0f-c711-4bac-96f8-5254b87278b6\") " pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-5kvt2" Oct 30 10:53:22.199804 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:22.199691 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"observability-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cc54d0f-c711-4bac-96f8-5254b87278b6-observability-operator-tls\") pod \"observability-operator-599c6f89d-5kvt2\" (UID: \"3cc54d0f-c711-4bac-96f8-5254b87278b6\") " pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-5kvt2" Oct 30 10:53:22.256609 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:22.256581 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-748bf975df-rlcvl"] Oct 30 10:53:22.260890 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:53:22.260860 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddecaf67e_662e_4f61_82ea_3f1f1ed143ff.slice/crio-2095aae8faedd318d6a00a75c6ae04afe927c9b9d9e0b1c29ccfef48da1fc154 WatchSource:0}: Error finding container 2095aae8faedd318d6a00a75c6ae04afe927c9b9d9e0b1c29ccfef48da1fc154: Status 404 returned error can't find the container with id 2095aae8faedd318d6a00a75c6ae04afe927c9b9d9e0b1c29ccfef48da1fc154 Oct 30 10:53:22.300266 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:22.300243 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mktm\" (UniqueName: \"kubernetes.io/projected/3cc54d0f-c711-4bac-96f8-5254b87278b6-kube-api-access-8mktm\") pod \"observability-operator-599c6f89d-5kvt2\" (UID: \"3cc54d0f-c711-4bac-96f8-5254b87278b6\") " pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-5kvt2" Oct 30 10:53:22.300370 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:22.300307 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"observability-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cc54d0f-c711-4bac-96f8-5254b87278b6-observability-operator-tls\") pod \"observability-operator-599c6f89d-5kvt2\" (UID: \"3cc54d0f-c711-4bac-96f8-5254b87278b6\") " pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-5kvt2" Oct 30 10:53:22.302598 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:22.302576 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"observability-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cc54d0f-c711-4bac-96f8-5254b87278b6-observability-operator-tls\") pod \"observability-operator-599c6f89d-5kvt2\" (UID: \"3cc54d0f-c711-4bac-96f8-5254b87278b6\") " pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-5kvt2" Oct 30 10:53:22.328046 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:22.328023 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mktm\" (UniqueName: \"kubernetes.io/projected/3cc54d0f-c711-4bac-96f8-5254b87278b6-kube-api-access-8mktm\") pod \"observability-operator-599c6f89d-5kvt2\" (UID: \"3cc54d0f-c711-4bac-96f8-5254b87278b6\") " pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-5kvt2" Oct 30 10:53:22.421521 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:22.421485 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-5kvt2" Oct 30 10:53:22.473642 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:22.473613 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-9wwp4" event={"ID":"61a6f0bb-fdca-4a0c-84b5-884a8dea32f8","Type":"ContainerStarted","Data":"ca4063227ac6580146afd27eecb9faa61583498f10c582ad2513a739957bdebc"} Oct 30 10:53:22.474924 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:22.474901 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-748bf975df-rlcvl" event={"ID":"decaf67e-662e-4f61-82ea-3f1f1ed143ff","Type":"ContainerStarted","Data":"2095aae8faedd318d6a00a75c6ae04afe927c9b9d9e0b1c29ccfef48da1fc154"} Oct 30 10:53:22.731154 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:22.730934 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/observability-operator-599c6f89d-5kvt2"] Oct 30 10:53:22.736012 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:53:22.735975 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc54d0f_c711_4bac_96f8_5254b87278b6.slice/crio-54b81505742b96b184af219b22e0c35e2af948819d3c70f529eb46206308665b WatchSource:0}: Error finding container 54b81505742b96b184af219b22e0c35e2af948819d3c70f529eb46206308665b: Status 404 returned error can't find the container with id 54b81505742b96b184af219b22e0c35e2af948819d3c70f529eb46206308665b Oct 30 10:53:23.482391 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:23.482310 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-5kvt2" event={"ID":"3cc54d0f-c711-4bac-96f8-5254b87278b6","Type":"ContainerStarted","Data":"54b81505742b96b184af219b22e0c35e2af948819d3c70f529eb46206308665b"} Oct 30 10:53:24.445914 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:24.445885 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-bd4f6797b-pjlxd" Oct 30 10:53:25.495804 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:25.495772 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-9wwp4" event={"ID":"61a6f0bb-fdca-4a0c-84b5-884a8dea32f8","Type":"ContainerStarted","Data":"c15817081af9b660e1a2bb4f2e606e6c6fefc2516419f1111a9d41ba790a0d2a"} Oct 30 10:53:25.585401 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:25.585346 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-9wwp4" podStartSLOduration=1.166314596 podStartE2EDuration="4.58532865s" totalImagesPullingTime="3.419014054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:53:25.584315355 +0000 UTC m=+496.085694841" watchObservedRunningTime="2026-10-30 10:53:25.58532865 +0000 UTC m=+496.086708136" Oct 30 10:53:26.256786 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:53:26.256751 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:53:26.501506 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:26.501469 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-748bf975df-rlcvl" event={"ID":"decaf67e-662e-4f61-82ea-3f1f1ed143ff","Type":"ContainerStarted","Data":"ec7ba10f7ee4067ed1b546a2ac1287b1047e0fe88b7f92a63244e37ad23715f3"} Oct 30 10:53:26.503231 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:26.503195 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-5kvt2" event={"ID":"3cc54d0f-c711-4bac-96f8-5254b87278b6","Type":"ContainerStarted","Data":"77e149b4f2aaf6f183f9c7411f73fc8643314bccaf1c4e138f7588f4d7a1e7d1"} Oct 30 10:53:26.503369 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:26.503258 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-5kvt2" Oct 30 10:53:26.504696 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:26.504661 4289 patch_prober.go:29] interesting pod/observability-operator-599c6f89d-5kvt2 container/operator namespace/openshift-cluster-observability-operator: Readiness probe status=failure output="Get \"http://10.134.0.26:8081/healthz\": dial tcp 10.134.0.26:8081: connect: connection refused" start-of-body= Oct 30 10:53:26.504810 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:26.504717 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-5kvt2" podUID="3cc54d0f-c711-4bac-96f8-5254b87278b6" containerName="operator" probeResult="failure" output="Get \"http://10.134.0.26:8081/healthz\": dial tcp 10.134.0.26:8081: connect: connection refused" Oct 30 10:53:26.597011 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:26.596913 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-748bf975df-rlcvl" podStartSLOduration=2.290453597 podStartE2EDuration="5.596896661s" totalImagesPullingTime="3.306443064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:53:26.586137004 +0000 UTC m=+497.087516501" watchObservedRunningTime="2026-10-30 10:53:26.596896661 +0000 UTC m=+497.098276146" Oct 30 10:53:26.653599 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:26.650618 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-5kvt2" podStartSLOduration=1.11587176 podStartE2EDuration="4.650601252s" totalImagesPullingTime="3.534729492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:53:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:53:26.64724324 +0000 UTC m=+497.148622725" watchObservedRunningTime="2026-10-30 10:53:26.650601252 +0000 UTC m=+497.151980739" Oct 30 10:53:27.510705 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:27.510675 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-5kvt2" Oct 30 10:53:38.389876 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:38.389840 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-867545c688-mjb5p"] Oct 30 10:53:38.393582 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:38.393558 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-867545c688-mjb5p" Oct 30 10:53:38.410511 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:38.410489 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-metrics\"" Oct 30 10:53:38.422533 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:38.422482 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-867545c688-mjb5p"] Oct 30 10:53:38.531680 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:38.531644 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/247ce84e-3c6c-4f42-af94-df147b3ce7a6-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-867545c688-mjb5p\" (UID: \"247ce84e-3c6c-4f42-af94-df147b3ce7a6\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-867545c688-mjb5p" Oct 30 10:53:38.531847 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:38.531704 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mtxh\" (UniqueName: \"kubernetes.io/projected/247ce84e-3c6c-4f42-af94-df147b3ce7a6-kube-api-access-6mtxh\") pod \"opentelemetry-operator-controller-manager-867545c688-mjb5p\" (UID: \"247ce84e-3c6c-4f42-af94-df147b3ce7a6\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-867545c688-mjb5p" Oct 30 10:53:38.632949 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:38.632913 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/247ce84e-3c6c-4f42-af94-df147b3ce7a6-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-867545c688-mjb5p\" (UID: \"247ce84e-3c6c-4f42-af94-df147b3ce7a6\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-867545c688-mjb5p" Oct 30 10:53:38.633151 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:38.632986 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mtxh\" (UniqueName: \"kubernetes.io/projected/247ce84e-3c6c-4f42-af94-df147b3ce7a6-kube-api-access-6mtxh\") pod \"opentelemetry-operator-controller-manager-867545c688-mjb5p\" (UID: \"247ce84e-3c6c-4f42-af94-df147b3ce7a6\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-867545c688-mjb5p" Oct 30 10:53:38.635272 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:38.635242 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/247ce84e-3c6c-4f42-af94-df147b3ce7a6-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-867545c688-mjb5p\" (UID: \"247ce84e-3c6c-4f42-af94-df147b3ce7a6\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-867545c688-mjb5p" Oct 30 10:53:38.674498 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:38.674421 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mtxh\" (UniqueName: \"kubernetes.io/projected/247ce84e-3c6c-4f42-af94-df147b3ce7a6-kube-api-access-6mtxh\") pod \"opentelemetry-operator-controller-manager-867545c688-mjb5p\" (UID: \"247ce84e-3c6c-4f42-af94-df147b3ce7a6\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-867545c688-mjb5p" Oct 30 10:53:38.704986 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:38.704953 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-867545c688-mjb5p" Oct 30 10:53:38.711398 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:38.711366 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:38.943110 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:38.943020 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:39.036446 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:39.036392 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-867545c688-mjb5p"] Oct 30 10:53:39.053397 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:53:39.053366 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247ce84e_3c6c_4f42_af94_df147b3ce7a6.slice/crio-15528c562832182183ec9a56b5275969985cc1fd2616432159eab7d7481e4988 WatchSource:0}: Error finding container 15528c562832182183ec9a56b5275969985cc1fd2616432159eab7d7481e4988: Status 404 returned error can't find the container with id 15528c562832182183ec9a56b5275969985cc1fd2616432159eab7d7481e4988 Oct 30 10:53:39.555861 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:39.555825 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-867545c688-mjb5p" event={"ID":"247ce84e-3c6c-4f42-af94-df147b3ce7a6","Type":"ContainerStarted","Data":"cbd3c0ca5e8bb2e24f0e367cd6bb7c30e40f4d9da553085d2bd9e9a6f15034dc"} Oct 30 10:53:39.556260 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:39.555884 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-867545c688-mjb5p" event={"ID":"247ce84e-3c6c-4f42-af94-df147b3ce7a6","Type":"ContainerStarted","Data":"15528c562832182183ec9a56b5275969985cc1fd2616432159eab7d7481e4988"} Oct 30 10:53:39.556260 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:39.555930 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-867545c688-mjb5p" Oct 30 10:53:39.604775 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:39.604724 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-867545c688-mjb5p" podStartSLOduration=1.604711542 podStartE2EDuration="1.604711542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:53:39.603985764 +0000 UTC m=+510.105365246" watchObservedRunningTime="2026-10-30 10:53:39.604711542 +0000 UTC m=+510.106091026" Oct 30 10:53:40.649191 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:40.649158 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-867545c688-mjb5p"] Oct 30 10:53:41.562701 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:41.562662 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-867545c688-mjb5p" podUID="247ce84e-3c6c-4f42-af94-df147b3ce7a6" containerName="manager" containerID="cri-o://cbd3c0ca5e8bb2e24f0e367cd6bb7c30e40f4d9da553085d2bd9e9a6f15034dc" gracePeriod=10 Oct 30 10:53:41.798259 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:41.798234 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-867545c688-mjb5p" Oct 30 10:53:41.958691 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:41.958611 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/247ce84e-3c6c-4f42-af94-df147b3ce7a6-opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/247ce84e-3c6c-4f42-af94-df147b3ce7a6-opentelemetry-operator-metrics-cert\") pod \"247ce84e-3c6c-4f42-af94-df147b3ce7a6\" (UID: \"247ce84e-3c6c-4f42-af94-df147b3ce7a6\") " Oct 30 10:53:41.958839 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:41.958692 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/247ce84e-3c6c-4f42-af94-df147b3ce7a6-kube-api-access-6mtxh\" (UniqueName: \"kubernetes.io/projected/247ce84e-3c6c-4f42-af94-df147b3ce7a6-kube-api-access-6mtxh\") pod \"247ce84e-3c6c-4f42-af94-df147b3ce7a6\" (UID: \"247ce84e-3c6c-4f42-af94-df147b3ce7a6\") " Oct 30 10:53:41.960827 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:41.960798 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247ce84e-3c6c-4f42-af94-df147b3ce7a6-kube-api-access-6mtxh" pod "247ce84e-3c6c-4f42-af94-df147b3ce7a6" (UID: "247ce84e-3c6c-4f42-af94-df147b3ce7a6"). InnerVolumeSpecName "kube-api-access-6mtxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:53:41.960941 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:41.960835 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/247ce84e-3c6c-4f42-af94-df147b3ce7a6-opentelemetry-operator-metrics-cert" pod "247ce84e-3c6c-4f42-af94-df147b3ce7a6" (UID: "247ce84e-3c6c-4f42-af94-df147b3ce7a6"). InnerVolumeSpecName "opentelemetry-operator-metrics-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:53:42.060029 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:42.059997 4289 reconciler_common.go:299] "Volume detached for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/247ce84e-3c6c-4f42-af94-df147b3ce7a6-opentelemetry-operator-metrics-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:53:42.060029 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:42.060022 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mtxh\" (UniqueName: \"kubernetes.io/projected/247ce84e-3c6c-4f42-af94-df147b3ce7a6-kube-api-access-6mtxh\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:53:42.568092 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:42.568056 4289 generic.go:356] "Generic (PLEG): container finished" podID="247ce84e-3c6c-4f42-af94-df147b3ce7a6" containerID="cbd3c0ca5e8bb2e24f0e367cd6bb7c30e40f4d9da553085d2bd9e9a6f15034dc" exitCode=0 Oct 30 10:53:42.568261 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:42.568115 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-867545c688-mjb5p" Oct 30 10:53:42.568261 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:42.568138 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-867545c688-mjb5p" event={"ID":"247ce84e-3c6c-4f42-af94-df147b3ce7a6","Type":"ContainerDied","Data":"cbd3c0ca5e8bb2e24f0e367cd6bb7c30e40f4d9da553085d2bd9e9a6f15034dc"} Oct 30 10:53:42.568261 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:42.568184 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-867545c688-mjb5p" event={"ID":"247ce84e-3c6c-4f42-af94-df147b3ce7a6","Type":"ContainerDied","Data":"15528c562832182183ec9a56b5275969985cc1fd2616432159eab7d7481e4988"} Oct 30 10:53:42.568261 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:42.568204 4289 scope.go:122] "RemoveContainer" containerID="cbd3c0ca5e8bb2e24f0e367cd6bb7c30e40f4d9da553085d2bd9e9a6f15034dc" Oct 30 10:53:42.576557 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:42.576539 4289 scope.go:122] "RemoveContainer" containerID="cbd3c0ca5e8bb2e24f0e367cd6bb7c30e40f4d9da553085d2bd9e9a6f15034dc" Oct 30 10:53:42.576759 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:53:42.576743 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbd3c0ca5e8bb2e24f0e367cd6bb7c30e40f4d9da553085d2bd9e9a6f15034dc\": container with ID starting with cbd3c0ca5e8bb2e24f0e367cd6bb7c30e40f4d9da553085d2bd9e9a6f15034dc not found: ID does not exist" containerID="cbd3c0ca5e8bb2e24f0e367cd6bb7c30e40f4d9da553085d2bd9e9a6f15034dc" Oct 30 10:53:42.576805 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:42.576767 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd3c0ca5e8bb2e24f0e367cd6bb7c30e40f4d9da553085d2bd9e9a6f15034dc"} err="failed to get container status \"cbd3c0ca5e8bb2e24f0e367cd6bb7c30e40f4d9da553085d2bd9e9a6f15034dc\": rpc error: code = NotFound desc = could not find container \"cbd3c0ca5e8bb2e24f0e367cd6bb7c30e40f4d9da553085d2bd9e9a6f15034dc\": container with ID starting with cbd3c0ca5e8bb2e24f0e367cd6bb7c30e40f4d9da553085d2bd9e9a6f15034dc not found: ID does not exist" Oct 30 10:53:42.636965 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:42.636942 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-867545c688-mjb5p"] Oct 30 10:53:42.640268 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:42.640251 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-867545c688-mjb5p"] Oct 30 10:53:43.624771 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:43.624742 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="247ce84e-3c6c-4f42-af94-df147b3ce7a6" path="/var/lib/kubelet/pods/247ce84e-3c6c-4f42-af94-df147b3ce7a6/volumes" Oct 30 10:53:51.907107 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:51.907063 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:52.077215 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:52.077181 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:52.367037 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:52.367003 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:52.537201 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:52.537167 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:53.050117 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:53.050081 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:54.328392 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:54.328357 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:54.518637 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:54.518603 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:55.739231 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:55.739201 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:55.917744 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:55.917705 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:56.123413 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:56.123331 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:56.257239 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:53:56.257202 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:53:56.296213 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:56.296187 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:56.793884 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:56.793845 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:56.979692 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:56.979664 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:58.794960 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:58.794907 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:53:59.003970 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:53:59.003883 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:00.875310 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:00.875268 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:01.053052 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:01.053008 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:02.996882 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:02.996849 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:03.236286 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:03.236247 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:13.823923 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:13.823885 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:13.992564 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:13.992526 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:18.181997 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.181957 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-ksvzt"] Oct 30 10:54:18.182499 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.182459 4289 cpu_manager.go:414] "containerMap: removing container" podUID="247ce84e-3c6c-4f42-af94-df147b3ce7a6" containerName="manager" Oct 30 10:54:18.182499 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.182478 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247ce84e-3c6c-4f42-af94-df147b3ce7a6" containerName="manager" Oct 30 10:54:18.182624 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.182608 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="247ce84e-3c6c-4f42-af94-df147b3ce7a6" containerName="manager" Oct 30 10:54:18.185140 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.185111 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:18.187502 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.187484 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-ksvzt" Oct 30 10:54:18.201872 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.201848 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 10:54:18.203308 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.203288 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-ksvzt"] Oct 30 10:54:18.213386 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.213367 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-ztbrl\"" Oct 30 10:54:18.213569 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.213551 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 10:54:18.213624 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.213593 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 10:54:18.213624 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.213607 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 10:54:18.236488 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.236464 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/aa553f37-d9cb-42eb-b410-5726396a3d6e-tls\") pod \"korrel8r-6644769df5-ksvzt\" (UID: \"aa553f37-d9cb-42eb-b410-5726396a3d6e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-ksvzt" Oct 30 10:54:18.236591 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.236508 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwr26\" (UniqueName: \"kubernetes.io/projected/aa553f37-d9cb-42eb-b410-5726396a3d6e-kube-api-access-jwr26\") pod \"korrel8r-6644769df5-ksvzt\" (UID: \"aa553f37-d9cb-42eb-b410-5726396a3d6e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-ksvzt" Oct 30 10:54:18.236591 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.236536 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa553f37-d9cb-42eb-b410-5726396a3d6e-config\") pod \"korrel8r-6644769df5-ksvzt\" (UID: \"aa553f37-d9cb-42eb-b410-5726396a3d6e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-ksvzt" Oct 30 10:54:18.336905 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.336874 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwr26\" (UniqueName: \"kubernetes.io/projected/aa553f37-d9cb-42eb-b410-5726396a3d6e-kube-api-access-jwr26\") pod \"korrel8r-6644769df5-ksvzt\" (UID: \"aa553f37-d9cb-42eb-b410-5726396a3d6e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-ksvzt" Oct 30 10:54:18.337074 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.336913 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa553f37-d9cb-42eb-b410-5726396a3d6e-config\") pod \"korrel8r-6644769df5-ksvzt\" (UID: \"aa553f37-d9cb-42eb-b410-5726396a3d6e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-ksvzt" Oct 30 10:54:18.337074 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.336953 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/aa553f37-d9cb-42eb-b410-5726396a3d6e-tls\") pod \"korrel8r-6644769df5-ksvzt\" (UID: \"aa553f37-d9cb-42eb-b410-5726396a3d6e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-ksvzt" Oct 30 10:54:18.337908 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.337880 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa553f37-d9cb-42eb-b410-5726396a3d6e-config\") pod \"korrel8r-6644769df5-ksvzt\" (UID: \"aa553f37-d9cb-42eb-b410-5726396a3d6e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-ksvzt" Oct 30 10:54:18.339330 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.339308 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/aa553f37-d9cb-42eb-b410-5726396a3d6e-tls\") pod \"korrel8r-6644769df5-ksvzt\" (UID: \"aa553f37-d9cb-42eb-b410-5726396a3d6e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-ksvzt" Oct 30 10:54:18.356724 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.356694 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:18.360192 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.360167 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwr26\" (UniqueName: \"kubernetes.io/projected/aa553f37-d9cb-42eb-b410-5726396a3d6e-kube-api-access-jwr26\") pod \"korrel8r-6644769df5-ksvzt\" (UID: \"aa553f37-d9cb-42eb-b410-5726396a3d6e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-ksvzt" Oct 30 10:54:18.498337 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.498297 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-ksvzt" Oct 30 10:54:18.520404 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.520367 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:18.791530 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:18.791502 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-ksvzt"] Oct 30 10:54:18.792077 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:54:18.792051 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa553f37_d9cb_42eb_b410_5726396a3d6e.slice/crio-0830e14a5b8befde609f1ea56ceaeca85f2473f5f55a355317996575d2ebee40 WatchSource:0}: Error finding container 0830e14a5b8befde609f1ea56ceaeca85f2473f5f55a355317996575d2ebee40: Status 404 returned error can't find the container with id 0830e14a5b8befde609f1ea56ceaeca85f2473f5f55a355317996575d2ebee40 Oct 30 10:54:19.043050 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:19.042966 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:19.090822 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:19.090786 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:54:19.093722 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:19.093700 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:19.113914 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:19.113887 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:54:19.143461 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:19.143421 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f6bd5f02-0220-4986-ab36-d5b0181f57dd-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f6bd5f02-0220-4986-ab36-d5b0181f57dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:19.143461 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:19.143463 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4l68\" (UniqueName: \"kubernetes.io/projected/f6bd5f02-0220-4986-ab36-d5b0181f57dd-kube-api-access-b4l68\") pod \"data-science-collector-collector-1\" (UID: \"f6bd5f02-0220-4986-ab36-d5b0181f57dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:19.143669 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:19.143595 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f6bd5f02-0220-4986-ab36-d5b0181f57dd-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f6bd5f02-0220-4986-ab36-d5b0181f57dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:19.188593 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:19.188569 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-61f7caf9\"" Oct 30 10:54:19.189780 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:19.189760 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-9gvds\"" Oct 30 10:54:19.189875 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:19.189798 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 10:54:19.208940 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:19.208908 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:19.244094 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:19.244074 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f6bd5f02-0220-4986-ab36-d5b0181f57dd-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f6bd5f02-0220-4986-ab36-d5b0181f57dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:19.244195 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:19.244111 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4l68\" (UniqueName: \"kubernetes.io/projected/f6bd5f02-0220-4986-ab36-d5b0181f57dd-kube-api-access-b4l68\") pod \"data-science-collector-collector-1\" (UID: \"f6bd5f02-0220-4986-ab36-d5b0181f57dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:19.244195 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:19.244175 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f6bd5f02-0220-4986-ab36-d5b0181f57dd-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f6bd5f02-0220-4986-ab36-d5b0181f57dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:19.244837 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:19.244815 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f6bd5f02-0220-4986-ab36-d5b0181f57dd-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f6bd5f02-0220-4986-ab36-d5b0181f57dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:19.246504 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:19.246487 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f6bd5f02-0220-4986-ab36-d5b0181f57dd-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f6bd5f02-0220-4986-ab36-d5b0181f57dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:19.266879 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:19.266858 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4l68\" (UniqueName: \"kubernetes.io/projected/f6bd5f02-0220-4986-ab36-d5b0181f57dd-kube-api-access-b4l68\") pod \"data-science-collector-collector-1\" (UID: \"f6bd5f02-0220-4986-ab36-d5b0181f57dd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:19.392120 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:19.392041 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:19.407875 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:19.407851 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:19.702209 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:54:19.702178 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6bd5f02_0220_4986_ab36_d5b0181f57dd.slice/crio-5fe4fc53ae4b10119ef46e6639be6ba4cfc55bcadd593de6a5846f73f3acdb8b WatchSource:0}: Error finding container 5fe4fc53ae4b10119ef46e6639be6ba4cfc55bcadd593de6a5846f73f3acdb8b: Status 404 returned error can't find the container with id 5fe4fc53ae4b10119ef46e6639be6ba4cfc55bcadd593de6a5846f73f3acdb8b Oct 30 10:54:19.702360 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:19.702317 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:54:19.718931 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:19.718900 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f6bd5f02-0220-4986-ab36-d5b0181f57dd","Type":"ContainerStarted","Data":"5fe4fc53ae4b10119ef46e6639be6ba4cfc55bcadd593de6a5846f73f3acdb8b"} Oct 30 10:54:19.720339 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:19.720312 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-ksvzt" event={"ID":"aa553f37-d9cb-42eb-b410-5726396a3d6e","Type":"ContainerStarted","Data":"0830e14a5b8befde609f1ea56ceaeca85f2473f5f55a355317996575d2ebee40"} Oct 30 10:54:20.074340 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:20.074306 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:20.243264 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:20.243226 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:20.409973 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:20.409892 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:21.545578 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:21.545536 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:21.707273 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:21.707212 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:21.731406 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:21.731370 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-ksvzt" event={"ID":"aa553f37-d9cb-42eb-b410-5726396a3d6e","Type":"ContainerStarted","Data":"2e42337411ec23b0d3a1ef6610486d37ccb9a54abc57675daa24a8ff3bc1202c"} Oct 30 10:54:21.731598 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:21.731471 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-ksvzt" Oct 30 10:54:21.772508 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:21.772461 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-ksvzt" podStartSLOduration=1.761937697 podStartE2EDuration="3.77244894s" totalImagesPullingTime="2.010511243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:54:21.770865367 +0000 UTC m=+552.272244886" watchObservedRunningTime="2026-10-30 10:54:21.77244894 +0000 UTC m=+552.273828421" Oct 30 10:54:22.195034 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:22.195004 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:22.355521 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:22.355485 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:23.297336 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:23.297303 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:23.727162 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:23.727131 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:54:23.727250 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:23.727212 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:54:24.746466 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:24.746417 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f6bd5f02-0220-4986-ab36-d5b0181f57dd","Type":"ContainerStarted","Data":"d657b472d9a97bb2d72bfb246172954c93262eb94faaa916c3ae4f7b594b50fc"} Oct 30 10:54:24.791169 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:24.791120 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.7693822670000001 podStartE2EDuration="5.791108594s" totalImagesPullingTime="4.021726327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:54:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:54:24.78972827 +0000 UTC m=+555.291107756" watchObservedRunningTime="2026-10-30 10:54:24.791108594 +0000 UTC m=+555.292488078" Oct 30 10:54:26.255795 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:54:26.255748 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:54:32.738934 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:32.738899 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-ksvzt" Oct 30 10:54:37.242853 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.242814 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-ksvzt"] Oct 30 10:54:37.243330 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.243029 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-ksvzt" podUID="aa553f37-d9cb-42eb-b410-5726396a3d6e" containerName="korrel8r" containerID="cri-o://2e42337411ec23b0d3a1ef6610486d37ccb9a54abc57675daa24a8ff3bc1202c" gracePeriod=30 Oct 30 10:54:37.490130 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.490107 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-ksvzt" Oct 30 10:54:37.600054 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.599967 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa553f37-d9cb-42eb-b410-5726396a3d6e-tls\" (UniqueName: \"kubernetes.io/secret/aa553f37-d9cb-42eb-b410-5726396a3d6e-tls\") pod \"aa553f37-d9cb-42eb-b410-5726396a3d6e\" (UID: \"aa553f37-d9cb-42eb-b410-5726396a3d6e\") " Oct 30 10:54:37.600238 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.600059 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa553f37-d9cb-42eb-b410-5726396a3d6e-kube-api-access-jwr26\" (UniqueName: \"kubernetes.io/projected/aa553f37-d9cb-42eb-b410-5726396a3d6e-kube-api-access-jwr26\") pod \"aa553f37-d9cb-42eb-b410-5726396a3d6e\" (UID: \"aa553f37-d9cb-42eb-b410-5726396a3d6e\") " Oct 30 10:54:37.600238 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.600151 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa553f37-d9cb-42eb-b410-5726396a3d6e-config\" (UniqueName: \"kubernetes.io/configmap/aa553f37-d9cb-42eb-b410-5726396a3d6e-config\") pod \"aa553f37-d9cb-42eb-b410-5726396a3d6e\" (UID: \"aa553f37-d9cb-42eb-b410-5726396a3d6e\") " Oct 30 10:54:37.600668 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.600639 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa553f37-d9cb-42eb-b410-5726396a3d6e-config" pod "aa553f37-d9cb-42eb-b410-5726396a3d6e" (UID: "aa553f37-d9cb-42eb-b410-5726396a3d6e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:54:37.602246 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.602220 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa553f37-d9cb-42eb-b410-5726396a3d6e-tls" pod "aa553f37-d9cb-42eb-b410-5726396a3d6e" (UID: "aa553f37-d9cb-42eb-b410-5726396a3d6e"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:54:37.602343 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.602324 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa553f37-d9cb-42eb-b410-5726396a3d6e-kube-api-access-jwr26" pod "aa553f37-d9cb-42eb-b410-5726396a3d6e" (UID: "aa553f37-d9cb-42eb-b410-5726396a3d6e"). InnerVolumeSpecName "kube-api-access-jwr26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:54:37.629162 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.629129 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:54:37.629379 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.629353 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="f6bd5f02-0220-4986-ab36-d5b0181f57dd" containerName="otc-container" containerID="cri-o://d657b472d9a97bb2d72bfb246172954c93262eb94faaa916c3ae4f7b594b50fc" gracePeriod=30 Oct 30 10:54:37.701587 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.701553 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwr26\" (UniqueName: \"kubernetes.io/projected/aa553f37-d9cb-42eb-b410-5726396a3d6e-kube-api-access-jwr26\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:54:37.701587 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.701586 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa553f37-d9cb-42eb-b410-5726396a3d6e-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:54:37.701779 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.701601 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/aa553f37-d9cb-42eb-b410-5726396a3d6e-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:54:37.728655 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.728628 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:37.801349 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.801320 4289 generic.go:356] "Generic (PLEG): container finished" podID="f6bd5f02-0220-4986-ab36-d5b0181f57dd" containerID="d657b472d9a97bb2d72bfb246172954c93262eb94faaa916c3ae4f7b594b50fc" exitCode=0 Oct 30 10:54:37.801471 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.801392 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f6bd5f02-0220-4986-ab36-d5b0181f57dd","Type":"ContainerDied","Data":"d657b472d9a97bb2d72bfb246172954c93262eb94faaa916c3ae4f7b594b50fc"} Oct 30 10:54:37.803780 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.803745 4289 generic.go:356] "Generic (PLEG): container finished" podID="aa553f37-d9cb-42eb-b410-5726396a3d6e" containerID="2e42337411ec23b0d3a1ef6610486d37ccb9a54abc57675daa24a8ff3bc1202c" exitCode=0 Oct 30 10:54:37.803952 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.803913 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-ksvzt" Oct 30 10:54:37.804727 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.803885 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-ksvzt" event={"ID":"aa553f37-d9cb-42eb-b410-5726396a3d6e","Type":"ContainerDied","Data":"2e42337411ec23b0d3a1ef6610486d37ccb9a54abc57675daa24a8ff3bc1202c"} Oct 30 10:54:37.804917 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.804894 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-ksvzt" event={"ID":"aa553f37-d9cb-42eb-b410-5726396a3d6e","Type":"ContainerDied","Data":"0830e14a5b8befde609f1ea56ceaeca85f2473f5f55a355317996575d2ebee40"} Oct 30 10:54:37.805138 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.805025 4289 scope.go:122] "RemoveContainer" containerID="2e42337411ec23b0d3a1ef6610486d37ccb9a54abc57675daa24a8ff3bc1202c" Oct 30 10:54:37.816827 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.816804 4289 scope.go:122] "RemoveContainer" containerID="2e42337411ec23b0d3a1ef6610486d37ccb9a54abc57675daa24a8ff3bc1202c" Oct 30 10:54:37.817092 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:54:37.817072 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e42337411ec23b0d3a1ef6610486d37ccb9a54abc57675daa24a8ff3bc1202c\": container with ID starting with 2e42337411ec23b0d3a1ef6610486d37ccb9a54abc57675daa24a8ff3bc1202c not found: ID does not exist" containerID="2e42337411ec23b0d3a1ef6610486d37ccb9a54abc57675daa24a8ff3bc1202c" Oct 30 10:54:37.817165 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.817099 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e42337411ec23b0d3a1ef6610486d37ccb9a54abc57675daa24a8ff3bc1202c"} err="failed to get container status \"2e42337411ec23b0d3a1ef6610486d37ccb9a54abc57675daa24a8ff3bc1202c\": rpc error: code = NotFound desc = could not find container \"2e42337411ec23b0d3a1ef6610486d37ccb9a54abc57675daa24a8ff3bc1202c\": container with ID starting with 2e42337411ec23b0d3a1ef6610486d37ccb9a54abc57675daa24a8ff3bc1202c not found: ID does not exist" Oct 30 10:54:37.853877 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.853821 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-ksvzt"] Oct 30 10:54:37.861560 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.861542 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:37.869811 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.869791 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-ksvzt"] Oct 30 10:54:37.899611 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:37.899585 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:38.003626 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:38.003598 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6bd5f02-0220-4986-ab36-d5b0181f57dd-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f6bd5f02-0220-4986-ab36-d5b0181f57dd-monitoring-tls-certs\") pod \"f6bd5f02-0220-4986-ab36-d5b0181f57dd\" (UID: \"f6bd5f02-0220-4986-ab36-d5b0181f57dd\") " Oct 30 10:54:38.003804 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:38.003636 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6bd5f02-0220-4986-ab36-d5b0181f57dd-otc-internal\" (UniqueName: \"kubernetes.io/configmap/f6bd5f02-0220-4986-ab36-d5b0181f57dd-otc-internal\") pod \"f6bd5f02-0220-4986-ab36-d5b0181f57dd\" (UID: \"f6bd5f02-0220-4986-ab36-d5b0181f57dd\") " Oct 30 10:54:38.003804 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:38.003696 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6bd5f02-0220-4986-ab36-d5b0181f57dd-kube-api-access-b4l68\" (UniqueName: \"kubernetes.io/projected/f6bd5f02-0220-4986-ab36-d5b0181f57dd-kube-api-access-b4l68\") pod \"f6bd5f02-0220-4986-ab36-d5b0181f57dd\" (UID: \"f6bd5f02-0220-4986-ab36-d5b0181f57dd\") " Oct 30 10:54:38.004043 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:38.004017 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6bd5f02-0220-4986-ab36-d5b0181f57dd-otc-internal" pod "f6bd5f02-0220-4986-ab36-d5b0181f57dd" (UID: "f6bd5f02-0220-4986-ab36-d5b0181f57dd"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:54:38.005638 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:38.005610 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6bd5f02-0220-4986-ab36-d5b0181f57dd-monitoring-tls-certs" pod "f6bd5f02-0220-4986-ab36-d5b0181f57dd" (UID: "f6bd5f02-0220-4986-ab36-d5b0181f57dd"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:54:38.005710 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:38.005695 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6bd5f02-0220-4986-ab36-d5b0181f57dd-kube-api-access-b4l68" pod "f6bd5f02-0220-4986-ab36-d5b0181f57dd" (UID: "f6bd5f02-0220-4986-ab36-d5b0181f57dd"). InnerVolumeSpecName "kube-api-access-b4l68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:54:38.105024 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:38.104953 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4l68\" (UniqueName: \"kubernetes.io/projected/f6bd5f02-0220-4986-ab36-d5b0181f57dd-kube-api-access-b4l68\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:54:38.105024 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:38.104980 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f6bd5f02-0220-4986-ab36-d5b0181f57dd-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:54:38.105024 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:38.104990 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f6bd5f02-0220-4986-ab36-d5b0181f57dd-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:54:38.167068 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:38.167032 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:38.334355 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:38.334317 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:38.809319 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:38.809290 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:38.809522 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:38.809295 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f6bd5f02-0220-4986-ab36-d5b0181f57dd","Type":"ContainerDied","Data":"5fe4fc53ae4b10119ef46e6639be6ba4cfc55bcadd593de6a5846f73f3acdb8b"} Oct 30 10:54:38.809522 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:38.809461 4289 scope.go:122] "RemoveContainer" containerID="d657b472d9a97bb2d72bfb246172954c93262eb94faaa916c3ae4f7b594b50fc" Oct 30 10:54:38.858341 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:38.858318 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:54:38.870560 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:38.870537 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:54:39.624158 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:39.624122 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa553f37-d9cb-42eb-b410-5726396a3d6e" path="/var/lib/kubelet/pods/aa553f37-d9cb-42eb-b410-5726396a3d6e/volumes" Oct 30 10:54:39.624530 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:39.624506 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6bd5f02-0220-4986-ab36-d5b0181f57dd" path="/var/lib/kubelet/pods/f6bd5f02-0220-4986-ab36-d5b0181f57dd/volumes" Oct 30 10:54:43.055560 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.055524 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:43.227075 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.227039 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:43.291755 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.291726 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8"] Oct 30 10:54:43.292052 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.292039 4289 cpu_manager.go:414] "containerMap: removing container" podUID="aa553f37-d9cb-42eb-b410-5726396a3d6e" containerName="korrel8r" Oct 30 10:54:43.292094 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.292054 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa553f37-d9cb-42eb-b410-5726396a3d6e" containerName="korrel8r" Oct 30 10:54:43.292094 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.292064 4289 cpu_manager.go:414] "containerMap: removing container" podUID="f6bd5f02-0220-4986-ab36-d5b0181f57dd" containerName="otc-container" Oct 30 10:54:43.292094 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.292069 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6bd5f02-0220-4986-ab36-d5b0181f57dd" containerName="otc-container" Oct 30 10:54:43.292185 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.292132 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa553f37-d9cb-42eb-b410-5726396a3d6e" containerName="korrel8r" Oct 30 10:54:43.292185 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.292138 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6bd5f02-0220-4986-ab36-d5b0181f57dd" containerName="otc-container" Oct 30 10:54:43.296502 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.296487 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" Oct 30 10:54:43.325021 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.324959 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 10:54:43.325195 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.324989 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 10:54:43.325308 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.325022 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 10:54:43.325308 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.325086 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-g7gbm\"" Oct 30 10:54:43.325308 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.325103 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 10:54:43.325308 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.325128 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 10:54:43.325308 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.325130 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 10:54:43.326345 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.326327 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8"] Oct 30 10:54:43.337000 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.336975 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks"] Oct 30 10:54:43.340410 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.340390 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" Oct 30 10:54:43.358747 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.358727 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks"] Oct 30 10:54:43.378784 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.378760 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9"] Oct 30 10:54:43.382440 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.382415 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" Oct 30 10:54:43.400559 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.400539 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9"] Oct 30 10:54:43.403179 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.403157 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:43.413458 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.413419 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 10:54:43.413562 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.413493 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 10:54:43.413621 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.413607 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-d9fl7\"" Oct 30 10:54:43.413938 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.413661 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-5gjhq\"" Oct 30 10:54:43.413938 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.413666 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 10:54:43.413938 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.413699 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 10:54:43.413938 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.413758 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 10:54:43.417134 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.417117 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8"] Oct 30 10:54:43.420231 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.420218 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8" Oct 30 10:54:43.439151 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.439130 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8"] Oct 30 10:54:43.452579 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.452557 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/073dc996-75bd-41d8-b531-93aaf2ba4b12-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks\" (UID: \"073dc996-75bd-41d8-b531-93aaf2ba4b12\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" Oct 30 10:54:43.452679 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.452584 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/742466b8-3c11-4a49-90bd-08aae667cdaf-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m6wz8\" (UID: \"742466b8-3c11-4a49-90bd-08aae667cdaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" Oct 30 10:54:43.452679 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.452615 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr8st\" (UniqueName: \"kubernetes.io/projected/742466b8-3c11-4a49-90bd-08aae667cdaf-kube-api-access-mr8st\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m6wz8\" (UID: \"742466b8-3c11-4a49-90bd-08aae667cdaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" Oct 30 10:54:43.452951 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.452680 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/742466b8-3c11-4a49-90bd-08aae667cdaf-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m6wz8\" (UID: \"742466b8-3c11-4a49-90bd-08aae667cdaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" Oct 30 10:54:43.452951 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.452787 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/073dc996-75bd-41d8-b531-93aaf2ba4b12-client-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks\" (UID: \"073dc996-75bd-41d8-b531-93aaf2ba4b12\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" Oct 30 10:54:43.452951 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.452840 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sdzk\" (UniqueName: \"kubernetes.io/projected/073dc996-75bd-41d8-b531-93aaf2ba4b12-kube-api-access-5sdzk\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks\" (UID: \"073dc996-75bd-41d8-b531-93aaf2ba4b12\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" Oct 30 10:54:43.452951 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.452868 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/742466b8-3c11-4a49-90bd-08aae667cdaf-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m6wz8\" (UID: \"742466b8-3c11-4a49-90bd-08aae667cdaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" Oct 30 10:54:43.453082 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.452997 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/742466b8-3c11-4a49-90bd-08aae667cdaf-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m6wz8\" (UID: \"742466b8-3c11-4a49-90bd-08aae667cdaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" Oct 30 10:54:43.453082 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.453047 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/073dc996-75bd-41d8-b531-93aaf2ba4b12-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks\" (UID: \"073dc996-75bd-41d8-b531-93aaf2ba4b12\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" Oct 30 10:54:43.453178 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.453093 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/073dc996-75bd-41d8-b531-93aaf2ba4b12-config\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks\" (UID: \"073dc996-75bd-41d8-b531-93aaf2ba4b12\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" Oct 30 10:54:43.487068 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.487044 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-h7gvj\"" Oct 30 10:54:43.487221 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.487100 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 10:54:43.513738 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.513714 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 10:54:43.554152 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.554127 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzdzj\" (UniqueName: \"kubernetes.io/projected/b4822533-f28b-4e97-959b-4e377442ddfd-kube-api-access-kzdzj\") pod \"data-science-thanos-querier-proxy-7b56895dd5-x8zb9\" (UID: \"b4822533-f28b-4e97-959b-4e377442ddfd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" Oct 30 10:54:43.554284 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.554162 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/073dc996-75bd-41d8-b531-93aaf2ba4b12-config\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks\" (UID: \"073dc996-75bd-41d8-b531-93aaf2ba4b12\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" Oct 30 10:54:43.554284 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.554186 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/073dc996-75bd-41d8-b531-93aaf2ba4b12-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks\" (UID: \"073dc996-75bd-41d8-b531-93aaf2ba4b12\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" Oct 30 10:54:43.554284 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.554278 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/742466b8-3c11-4a49-90bd-08aae667cdaf-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m6wz8\" (UID: \"742466b8-3c11-4a49-90bd-08aae667cdaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" Oct 30 10:54:43.554412 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.554362 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4a5f36fe-3925-420e-9a5f-1e2d1e70a78e-tls\") pod \"korrel8r-5cbcc89b8f-pvcx8\" (UID: \"4a5f36fe-3925-420e-9a5f-1e2d1e70a78e\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8" Oct 30 10:54:43.554412 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.554385 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr8st\" (UniqueName: \"kubernetes.io/projected/742466b8-3c11-4a49-90bd-08aae667cdaf-kube-api-access-mr8st\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m6wz8\" (UID: \"742466b8-3c11-4a49-90bd-08aae667cdaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" Oct 30 10:54:43.554544 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.554475 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/742466b8-3c11-4a49-90bd-08aae667cdaf-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m6wz8\" (UID: \"742466b8-3c11-4a49-90bd-08aae667cdaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" Oct 30 10:54:43.554544 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.554522 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/073dc996-75bd-41d8-b531-93aaf2ba4b12-client-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks\" (UID: \"073dc996-75bd-41d8-b531-93aaf2ba4b12\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" Oct 30 10:54:43.554649 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.554551 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b4822533-f28b-4e97-959b-4e377442ddfd-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-x8zb9\" (UID: \"b4822533-f28b-4e97-959b-4e377442ddfd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" Oct 30 10:54:43.554649 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.554590 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sdzk\" (UniqueName: \"kubernetes.io/projected/073dc996-75bd-41d8-b531-93aaf2ba4b12-kube-api-access-5sdzk\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks\" (UID: \"073dc996-75bd-41d8-b531-93aaf2ba4b12\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" Oct 30 10:54:43.554649 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.554618 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/742466b8-3c11-4a49-90bd-08aae667cdaf-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m6wz8\" (UID: \"742466b8-3c11-4a49-90bd-08aae667cdaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" Oct 30 10:54:43.554797 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.554694 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/742466b8-3c11-4a49-90bd-08aae667cdaf-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m6wz8\" (UID: \"742466b8-3c11-4a49-90bd-08aae667cdaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" Oct 30 10:54:43.554797 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.554728 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhrc4\" (UniqueName: \"kubernetes.io/projected/4a5f36fe-3925-420e-9a5f-1e2d1e70a78e-kube-api-access-mhrc4\") pod \"korrel8r-5cbcc89b8f-pvcx8\" (UID: \"4a5f36fe-3925-420e-9a5f-1e2d1e70a78e\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8" Oct 30 10:54:43.554971 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.554942 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4822533-f28b-4e97-959b-4e377442ddfd-config\") pod \"data-science-thanos-querier-proxy-7b56895dd5-x8zb9\" (UID: \"b4822533-f28b-4e97-959b-4e377442ddfd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" Oct 30 10:54:43.555048 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.554988 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a5f36fe-3925-420e-9a5f-1e2d1e70a78e-config\") pod \"korrel8r-5cbcc89b8f-pvcx8\" (UID: \"4a5f36fe-3925-420e-9a5f-1e2d1e70a78e\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8" Oct 30 10:54:43.555048 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.555029 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/073dc996-75bd-41d8-b531-93aaf2ba4b12-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks\" (UID: \"073dc996-75bd-41d8-b531-93aaf2ba4b12\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" Oct 30 10:54:43.555162 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:54:43.555045 4289 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 30 10:54:43.555162 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.555071 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/073dc996-75bd-41d8-b531-93aaf2ba4b12-config\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks\" (UID: \"073dc996-75bd-41d8-b531-93aaf2ba4b12\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" Oct 30 10:54:43.555162 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.555074 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/073dc996-75bd-41d8-b531-93aaf2ba4b12-client-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks\" (UID: \"073dc996-75bd-41d8-b531-93aaf2ba4b12\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" Oct 30 10:54:43.555162 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:54:43.555105 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/742466b8-3c11-4a49-90bd-08aae667cdaf-prometheus-client-cert podName:742466b8-3c11-4a49-90bd-08aae667cdaf nodeName:}" failed. No retries permitted until 2026-10-30 10:54:44.055085313 +0000 UTC m=+574.556464775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/742466b8-3c11-4a49-90bd-08aae667cdaf-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" (UID: "742466b8-3c11-4a49-90bd-08aae667cdaf") : secret "prometheus-operated-tls" not found Oct 30 10:54:43.555379 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.555228 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/742466b8-3c11-4a49-90bd-08aae667cdaf-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m6wz8\" (UID: \"742466b8-3c11-4a49-90bd-08aae667cdaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" Oct 30 10:54:43.555590 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.555569 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/073dc996-75bd-41d8-b531-93aaf2ba4b12-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks\" (UID: \"073dc996-75bd-41d8-b531-93aaf2ba4b12\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" Oct 30 10:54:43.556836 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.556817 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/073dc996-75bd-41d8-b531-93aaf2ba4b12-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks\" (UID: \"073dc996-75bd-41d8-b531-93aaf2ba4b12\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" Oct 30 10:54:43.557389 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.557371 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/742466b8-3c11-4a49-90bd-08aae667cdaf-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m6wz8\" (UID: \"742466b8-3c11-4a49-90bd-08aae667cdaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" Oct 30 10:54:43.557492 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.557473 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/742466b8-3c11-4a49-90bd-08aae667cdaf-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m6wz8\" (UID: \"742466b8-3c11-4a49-90bd-08aae667cdaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" Oct 30 10:54:43.577846 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.577789 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr8st\" (UniqueName: \"kubernetes.io/projected/742466b8-3c11-4a49-90bd-08aae667cdaf-kube-api-access-mr8st\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m6wz8\" (UID: \"742466b8-3c11-4a49-90bd-08aae667cdaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" Oct 30 10:54:43.577977 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.577861 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sdzk\" (UniqueName: \"kubernetes.io/projected/073dc996-75bd-41d8-b531-93aaf2ba4b12-kube-api-access-5sdzk\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks\" (UID: \"073dc996-75bd-41d8-b531-93aaf2ba4b12\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" Oct 30 10:54:43.655939 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.655908 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhrc4\" (UniqueName: \"kubernetes.io/projected/4a5f36fe-3925-420e-9a5f-1e2d1e70a78e-kube-api-access-mhrc4\") pod \"korrel8r-5cbcc89b8f-pvcx8\" (UID: \"4a5f36fe-3925-420e-9a5f-1e2d1e70a78e\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8" Oct 30 10:54:43.655939 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.655940 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4822533-f28b-4e97-959b-4e377442ddfd-config\") pod \"data-science-thanos-querier-proxy-7b56895dd5-x8zb9\" (UID: \"b4822533-f28b-4e97-959b-4e377442ddfd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" Oct 30 10:54:43.656165 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.655958 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a5f36fe-3925-420e-9a5f-1e2d1e70a78e-config\") pod \"korrel8r-5cbcc89b8f-pvcx8\" (UID: \"4a5f36fe-3925-420e-9a5f-1e2d1e70a78e\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8" Oct 30 10:54:43.656165 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.655987 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzdzj\" (UniqueName: \"kubernetes.io/projected/b4822533-f28b-4e97-959b-4e377442ddfd-kube-api-access-kzdzj\") pod \"data-science-thanos-querier-proxy-7b56895dd5-x8zb9\" (UID: \"b4822533-f28b-4e97-959b-4e377442ddfd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" Oct 30 10:54:43.656165 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.656011 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4a5f36fe-3925-420e-9a5f-1e2d1e70a78e-tls\") pod \"korrel8r-5cbcc89b8f-pvcx8\" (UID: \"4a5f36fe-3925-420e-9a5f-1e2d1e70a78e\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8" Oct 30 10:54:43.656165 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.656033 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b4822533-f28b-4e97-959b-4e377442ddfd-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-x8zb9\" (UID: \"b4822533-f28b-4e97-959b-4e377442ddfd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" Oct 30 10:54:43.656364 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.656216 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" Oct 30 10:54:43.656842 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.656818 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4822533-f28b-4e97-959b-4e377442ddfd-config\") pod \"data-science-thanos-querier-proxy-7b56895dd5-x8zb9\" (UID: \"b4822533-f28b-4e97-959b-4e377442ddfd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" Oct 30 10:54:43.656955 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.656859 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a5f36fe-3925-420e-9a5f-1e2d1e70a78e-config\") pod \"korrel8r-5cbcc89b8f-pvcx8\" (UID: \"4a5f36fe-3925-420e-9a5f-1e2d1e70a78e\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8" Oct 30 10:54:43.658317 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.658294 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4a5f36fe-3925-420e-9a5f-1e2d1e70a78e-tls\") pod \"korrel8r-5cbcc89b8f-pvcx8\" (UID: \"4a5f36fe-3925-420e-9a5f-1e2d1e70a78e\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8" Oct 30 10:54:43.658394 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.658324 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b4822533-f28b-4e97-959b-4e377442ddfd-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-x8zb9\" (UID: \"b4822533-f28b-4e97-959b-4e377442ddfd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" Oct 30 10:54:43.678888 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.678865 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhrc4\" (UniqueName: \"kubernetes.io/projected/4a5f36fe-3925-420e-9a5f-1e2d1e70a78e-kube-api-access-mhrc4\") pod \"korrel8r-5cbcc89b8f-pvcx8\" (UID: \"4a5f36fe-3925-420e-9a5f-1e2d1e70a78e\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8" Oct 30 10:54:43.679004 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.678903 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzdzj\" (UniqueName: \"kubernetes.io/projected/b4822533-f28b-4e97-959b-4e377442ddfd-kube-api-access-kzdzj\") pod \"data-science-thanos-querier-proxy-7b56895dd5-x8zb9\" (UID: \"b4822533-f28b-4e97-959b-4e377442ddfd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" Oct 30 10:54:43.699242 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.699212 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" Oct 30 10:54:43.732857 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.732771 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8" Oct 30 10:54:43.949324 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.949292 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks"] Oct 30 10:54:43.949820 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:54:43.949792 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod073dc996_75bd_41d8_b531_93aaf2ba4b12.slice/crio-ce82f2007189e6846d3f8eb10e82d7284524ba81e7dd600dff6a323006f7deb0 WatchSource:0}: Error finding container ce82f2007189e6846d3f8eb10e82d7284524ba81e7dd600dff6a323006f7deb0: Status 404 returned error can't find the container with id ce82f2007189e6846d3f8eb10e82d7284524ba81e7dd600dff6a323006f7deb0 Oct 30 10:54:43.954555 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.954522 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:43.998801 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:43.998775 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9"] Oct 30 10:54:43.998965 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:54:43.998947 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4822533_f28b_4e97_959b_4e377442ddfd.slice/crio-4b0a0400db1bd4d6ed89214120fbfc5b42aa0d2e6da35614ba3068f01fbbdd4b WatchSource:0}: Error finding container 4b0a0400db1bd4d6ed89214120fbfc5b42aa0d2e6da35614ba3068f01fbbdd4b: Status 404 returned error can't find the container with id 4b0a0400db1bd4d6ed89214120fbfc5b42aa0d2e6da35614ba3068f01fbbdd4b Oct 30 10:54:44.036869 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.036848 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8"] Oct 30 10:54:44.037331 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:54:44.037315 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a5f36fe_3925_420e_9a5f_1e2d1e70a78e.slice/crio-705669658f2fb318c485aced9e38f7fbd3bd104271335dfbeaa0c04b467f52a8 WatchSource:0}: Error finding container 705669658f2fb318c485aced9e38f7fbd3bd104271335dfbeaa0c04b467f52a8: Status 404 returned error can't find the container with id 705669658f2fb318c485aced9e38f7fbd3bd104271335dfbeaa0c04b467f52a8 Oct 30 10:54:44.059703 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.059678 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/742466b8-3c11-4a49-90bd-08aae667cdaf-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m6wz8\" (UID: \"742466b8-3c11-4a49-90bd-08aae667cdaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" Oct 30 10:54:44.061796 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.061779 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/742466b8-3c11-4a49-90bd-08aae667cdaf-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m6wz8\" (UID: \"742466b8-3c11-4a49-90bd-08aae667cdaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" Oct 30 10:54:44.115791 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.115755 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:44.208934 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.208850 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" Oct 30 10:54:44.278646 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.278612 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:44.436816 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.436782 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:44.501781 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.501753 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8"] Oct 30 10:54:44.502500 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:54:44.502472 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod742466b8_3c11_4a49_90bd_08aae667cdaf.slice/crio-8d6a0db23ca893f4a9f94802ad86443b8d7a3967e88cbe447cb07d974dd5fcd1 WatchSource:0}: Error finding container 8d6a0db23ca893f4a9f94802ad86443b8d7a3967e88cbe447cb07d974dd5fcd1: Status 404 returned error can't find the container with id 8d6a0db23ca893f4a9f94802ad86443b8d7a3967e88cbe447cb07d974dd5fcd1 Oct 30 10:54:44.828257 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.828179 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:54:44.832330 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.832306 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:44.850625 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.850458 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" event={"ID":"073dc996-75bd-41d8-b531-93aaf2ba4b12","Type":"ContainerStarted","Data":"ce82f2007189e6846d3f8eb10e82d7284524ba81e7dd600dff6a323006f7deb0"} Oct 30 10:54:44.855602 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.855578 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:54:44.857997 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.857967 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" event={"ID":"b4822533-f28b-4e97-959b-4e377442ddfd","Type":"ContainerStarted","Data":"4b0a0400db1bd4d6ed89214120fbfc5b42aa0d2e6da35614ba3068f01fbbdd4b"} Oct 30 10:54:44.861488 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.860793 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8" event={"ID":"4a5f36fe-3925-420e-9a5f-1e2d1e70a78e","Type":"ContainerStarted","Data":"266399455f3ea9c4b4a38815d8756bc7091879d59d7971ba2cb2661f48f83ae5"} Oct 30 10:54:44.861488 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.860844 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8" event={"ID":"4a5f36fe-3925-420e-9a5f-1e2d1e70a78e","Type":"ContainerStarted","Data":"705669658f2fb318c485aced9e38f7fbd3bd104271335dfbeaa0c04b467f52a8"} Oct 30 10:54:44.861488 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.861414 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8" Oct 30 10:54:44.862806 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.862747 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" event={"ID":"742466b8-3c11-4a49-90bd-08aae667cdaf","Type":"ContainerStarted","Data":"8d6a0db23ca893f4a9f94802ad86443b8d7a3967e88cbe447cb07d974dd5fcd1"} Oct 30 10:54:44.867791 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.866698 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 10:54:44.867791 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.867066 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 10:54:44.867791 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.867321 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8vrjb\"" Oct 30 10:54:44.867791 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.867416 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 10:54:44.874239 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.874217 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-e4cadd38\"" Oct 30 10:54:44.906752 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.906721 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:44.921031 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.920857 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8" podStartSLOduration=1.920838395 podStartE2EDuration="1.920838395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:54:44.918814143 +0000 UTC m=+575.420193846" watchObservedRunningTime="2026-10-30 10:54:44.920838395 +0000 UTC m=+575.422217881" Oct 30 10:54:44.975207 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.975170 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:44.975360 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.975213 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:44.975360 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.975245 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:44.975360 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.975301 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm49g\" (UniqueName: \"kubernetes.io/projected/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-kube-api-access-jm49g\") pod \"data-science-collector-collector-1\" (UID: \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:44.975360 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:44.975327 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:45.076285 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:45.076245 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:45.076783 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:45.076355 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:45.076783 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:45.076385 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:45.076783 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:45.076420 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:45.076783 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:45.076502 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm49g\" (UniqueName: \"kubernetes.io/projected/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-kube-api-access-jm49g\") pod \"data-science-collector-collector-1\" (UID: \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:45.078102 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:45.078073 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:45.078214 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:54:45.078176 4289 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-client-cert: secret "data-science-collector-ta-client-cert" not found Oct 30 10:54:45.078300 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:54:45.078234 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-data-science-collector-ta-client-cert podName:ca2cf81c-da8d-4ac7-b560-4e264cd006bc nodeName:}" failed. No retries permitted until 2026-10-30 10:54:45.57821622 +0000 UTC m=+576.079595689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-client-cert" (UniqueName: "kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-data-science-collector-ta-client-cert") pod "data-science-collector-collector-1" (UID: "ca2cf81c-da8d-4ac7-b560-4e264cd006bc") : secret "data-science-collector-ta-client-cert" not found Oct 30 10:54:45.082957 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:45.082871 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:45.083387 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:45.083366 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:45.086230 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:45.086206 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:45.108677 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:45.108650 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm49g\" (UniqueName: \"kubernetes.io/projected/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-kube-api-access-jm49g\") pod \"data-science-collector-collector-1\" (UID: \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:45.581908 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:45.581868 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:45.582105 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:54:45.582030 4289 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-client-cert: secret "data-science-collector-ta-client-cert" not found Oct 30 10:54:45.582105 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:54:45.582095 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-data-science-collector-ta-client-cert podName:ca2cf81c-da8d-4ac7-b560-4e264cd006bc nodeName:}" failed. No retries permitted until 2026-10-30 10:54:46.582075071 +0000 UTC m=+577.083454534 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-client-cert" (UniqueName: "kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-data-science-collector-ta-client-cert") pod "data-science-collector-collector-1" (UID: "ca2cf81c-da8d-4ac7-b560-4e264cd006bc") : secret "data-science-collector-ta-client-cert" not found Oct 30 10:54:46.589639 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:46.589597 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:46.609130 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:46.609100 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:46.653147 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:46.653113 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:54:46.965982 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:46.965896 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:54:46.966626 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:54:46.966557 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2cf81c_da8d_4ac7_b560_4e264cd006bc.slice/crio-0c3219f3a26a33dde9268f0050923461392bd2d94e3e9fd047b89fd1fdbfcb90 WatchSource:0}: Error finding container 0c3219f3a26a33dde9268f0050923461392bd2d94e3e9fd047b89fd1fdbfcb90: Status 404 returned error can't find the container with id 0c3219f3a26a33dde9268f0050923461392bd2d94e3e9fd047b89fd1fdbfcb90 Oct 30 10:54:46.971853 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:46.971745 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:54:46.971853 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:46.971830 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:54:47.575045 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:47.575016 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:47.823221 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:47.823133 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:47.883210 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:47.883180 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" event={"ID":"073dc996-75bd-41d8-b531-93aaf2ba4b12","Type":"ContainerStarted","Data":"142073b844cc022f6e0c6f06e3a3818e9720584cb5b8762715e09fc3503f2c95"} Oct 30 10:54:47.883373 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:47.883256 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" event={"ID":"073dc996-75bd-41d8-b531-93aaf2ba4b12","Type":"ContainerStarted","Data":"bb4657afdd237952f25127416e3df9bd958c4868c437255a98c7b4b4b575a96f"} Oct 30 10:54:47.885520 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:47.885469 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" event={"ID":"b4822533-f28b-4e97-959b-4e377442ddfd","Type":"ContainerStarted","Data":"6a3c9d6f35ac1b0ed4912648c65edd9bac75bbcf3336d76ea90ac184f45eb826"} Oct 30 10:54:47.885666 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:47.885574 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" event={"ID":"b4822533-f28b-4e97-959b-4e377442ddfd","Type":"ContainerStarted","Data":"51fbc7c5dc57675c0ef2dd867a9011b13758074dc11f33c974b987186e2128d8"} Oct 30 10:54:47.887224 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:47.887196 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ca2cf81c-da8d-4ac7-b560-4e264cd006bc","Type":"ContainerStarted","Data":"f70f1adfb3b8a68ae73d90425219835abd6e26f8c4814e89caeaacd4eeefc735"} Oct 30 10:54:47.887349 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:47.887247 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ca2cf81c-da8d-4ac7-b560-4e264cd006bc","Type":"ContainerStarted","Data":"0c3219f3a26a33dde9268f0050923461392bd2d94e3e9fd047b89fd1fdbfcb90"} Oct 30 10:54:47.929228 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:47.929181 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=3.929164459 podStartE2EDuration="3.929164459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:54:47.927808381 +0000 UTC m=+578.429187868" watchObservedRunningTime="2026-10-30 10:54:47.929164459 +0000 UTC m=+578.430543945" Oct 30 10:54:47.988345 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:47.988253 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:48.199971 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.199936 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 10:54:48.203581 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.203555 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:54:48.217574 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.217532 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 10:54:48.218447 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.217925 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 10:54:48.218447 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.217961 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 10:54:48.229505 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.229474 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 10:54:48.229630 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.229605 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-vfhvl\"" Oct 30 10:54:48.231339 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.231282 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:48.233049 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.233023 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 10:54:48.306195 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.306157 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/75a3f2af-e799-4bfc-bc43-f40ab46c0913-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:54:48.306343 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.306238 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75a3f2af-e799-4bfc-bc43-f40ab46c0913-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:54:48.306343 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.306284 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75a3f2af-e799-4bfc-bc43-f40ab46c0913-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:54:48.306343 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.306315 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75a3f2af-e799-4bfc-bc43-f40ab46c0913-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:54:48.306531 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.306398 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/75a3f2af-e799-4bfc-bc43-f40ab46c0913-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:54:48.306531 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.306493 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75a3f2af-e799-4bfc-bc43-f40ab46c0913-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:54:48.306626 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.306539 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbx6p\" (UniqueName: \"kubernetes.io/projected/75a3f2af-e799-4bfc-bc43-f40ab46c0913-kube-api-access-dbx6p\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:54:48.407384 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.407350 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75a3f2af-e799-4bfc-bc43-f40ab46c0913-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:54:48.407565 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.407394 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbx6p\" (UniqueName: \"kubernetes.io/projected/75a3f2af-e799-4bfc-bc43-f40ab46c0913-kube-api-access-dbx6p\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:54:48.407565 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.407487 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/75a3f2af-e799-4bfc-bc43-f40ab46c0913-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:54:48.407565 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.407528 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75a3f2af-e799-4bfc-bc43-f40ab46c0913-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:54:48.407565 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.407560 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75a3f2af-e799-4bfc-bc43-f40ab46c0913-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:54:48.407851 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.407589 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75a3f2af-e799-4bfc-bc43-f40ab46c0913-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:54:48.407851 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.407675 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/75a3f2af-e799-4bfc-bc43-f40ab46c0913-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:54:48.408227 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.408178 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/75a3f2af-e799-4bfc-bc43-f40ab46c0913-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:54:48.412367 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.410972 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75a3f2af-e799-4bfc-bc43-f40ab46c0913-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:54:48.412367 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.411140 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75a3f2af-e799-4bfc-bc43-f40ab46c0913-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:54:48.412367 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.411727 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75a3f2af-e799-4bfc-bc43-f40ab46c0913-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:54:48.412367 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.411769 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/75a3f2af-e799-4bfc-bc43-f40ab46c0913-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:54:48.412905 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.412829 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:48.415110 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.415085 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75a3f2af-e799-4bfc-bc43-f40ab46c0913-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:54:48.435015 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.434988 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbx6p\" (UniqueName: \"kubernetes.io/projected/75a3f2af-e799-4bfc-bc43-f40ab46c0913-kube-api-access-dbx6p\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:54:48.523328 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.523290 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:54:48.579573 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.579529 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:48.801976 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.801878 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:48.851985 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:54:48.851943 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a3f2af_e799_4bfc_bc43_f40ab46c0913.slice/crio-843fa932192557a70ef67f6b91568dec72185fa0750db6b4b4d0f3674cc3a7f2 WatchSource:0}: Error finding container 843fa932192557a70ef67f6b91568dec72185fa0750db6b4b4d0f3674cc3a7f2: Status 404 returned error can't find the container with id 843fa932192557a70ef67f6b91568dec72185fa0750db6b4b4d0f3674cc3a7f2 Oct 30 10:54:48.852302 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.852068 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 10:54:48.894146 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.894118 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" event={"ID":"073dc996-75bd-41d8-b531-93aaf2ba4b12","Type":"ContainerStarted","Data":"9c1f27e1d764132f027948096dc8902f2b3cf898f1e6a18072627826fb505085"} Oct 30 10:54:48.896160 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.896130 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" event={"ID":"b4822533-f28b-4e97-959b-4e377442ddfd","Type":"ContainerStarted","Data":"b2386b3362d81b4da34ca01a5d45d94caa39807da09c80e94564bf55498256f0"} Oct 30 10:54:48.897362 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.897340 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"75a3f2af-e799-4bfc-bc43-f40ab46c0913","Type":"ContainerStarted","Data":"843fa932192557a70ef67f6b91568dec72185fa0750db6b4b4d0f3674cc3a7f2"} Oct 30 10:54:48.898675 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.898651 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" event={"ID":"742466b8-3c11-4a49-90bd-08aae667cdaf","Type":"ContainerStarted","Data":"ae6a182049bc84539c34da26bceed11b915379a0ca77c23f8bd562000d04d8aa"} Oct 30 10:54:48.898755 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.898741 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" Oct 30 10:54:48.953200 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:48.953153 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" podStartSLOduration=1.427893483 podStartE2EDuration="5.953137054s" totalImagesPullingTime="4.525243571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:54:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:54:48.95303379 +0000 UTC m=+579.454413289" watchObservedRunningTime="2026-10-30 10:54:48.953137054 +0000 UTC m=+579.454516540" Oct 30 10:54:49.019582 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:49.010842 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" podStartSLOduration=2.983067604 podStartE2EDuration="6.010824651s" totalImagesPullingTime="3.027757047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:54:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:54:49.004535357 +0000 UTC m=+579.505914834" watchObservedRunningTime="2026-10-30 10:54:49.010824651 +0000 UTC m=+579.512204137" Oct 30 10:54:49.060501 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:49.060335 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" podStartSLOduration=1.692732426 podStartE2EDuration="6.06031806s" totalImagesPullingTime="4.367585634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:54:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:54:49.058124307 +0000 UTC m=+579.559503792" watchObservedRunningTime="2026-10-30 10:54:49.06031806 +0000 UTC m=+579.561697544" Oct 30 10:54:50.811879 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:50.811839 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:50.968134 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:50.968097 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:51.538315 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.538275 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 10:54:51.545294 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.545263 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.556179 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.556153 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 10:54:51.599278 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.599246 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 10:54:51.599278 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.599252 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 10:54:51.599519 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.599293 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 10:54:51.599519 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.599307 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 10:54:51.599519 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.599311 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 10:54:51.599519 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.599334 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 10:54:51.599519 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.599336 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 10:54:51.599519 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.599386 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-jh8zq\"" Oct 30 10:54:51.599710 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.599521 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 10:54:51.616747 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.616720 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:51.653340 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.653313 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.653500 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.653361 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.653500 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.653379 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.653500 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.653478 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.653620 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.653532 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.653620 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.653577 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.653620 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.653602 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.653741 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.653629 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.653741 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.653646 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.653741 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.653735 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.653838 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.653778 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.653838 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.653793 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nkpq\" (UniqueName: \"kubernetes.io/projected/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-kube-api-access-4nkpq\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.754567 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.754530 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.754567 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.754565 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nkpq\" (UniqueName: \"kubernetes.io/projected/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-kube-api-access-4nkpq\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.754752 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.754589 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.754752 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.754626 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.754831 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.754804 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.754867 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.754856 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.754947 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.754905 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.755009 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.754977 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.755067 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.755006 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.755067 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.755043 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.755171 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.755071 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.755171 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.755125 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.755953 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.755500 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.755953 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.755906 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.756132 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.755991 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.757718 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.757697 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.757831 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.757808 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.757900 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.757870 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.757941 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.757915 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.757941 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.757927 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.758460 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.758423 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.758644 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.758628 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.759746 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.759725 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.776869 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.776846 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nkpq\" (UniqueName: \"kubernetes.io/projected/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-kube-api-access-4nkpq\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:51.781130 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.781105 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:51.912753 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:51.912725 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"75a3f2af-e799-4bfc-bc43-f40ab46c0913","Type":"ContainerStarted","Data":"77ba23d9377c25b188c49d964a594a298d1ff3366744dc2b698cc841933ca93d"} Oct 30 10:54:54.427101 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:54.427068 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:54:54.906308 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:54.906226 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" Oct 30 10:54:55.283705 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:55.283675 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:55.311442 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:55.311397 4289 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:55.384866 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:55.384826 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:55.385039 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:55.384995 4289 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:55.398869 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:55.398841 4289 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") DevicePath \"csi-ea06c66241a0a003a5daff5d9fc602f40e53768b00b104c5e4118b92d558c090\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:55.580364 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:55.580331 4289 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7342da0537aa528cc1545d2018ac3fd0382dfe4f670067e58cf8ba34bb891174/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:55.604615 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:55.604591 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:55.766474 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:55.766409 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:54:55.934390 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:55.934359 4289 generic.go:356] "Generic (PLEG): container finished" podID="75a3f2af-e799-4bfc-bc43-f40ab46c0913" containerID="77ba23d9377c25b188c49d964a594a298d1ff3366744dc2b698cc841933ca93d" exitCode=0 Oct 30 10:54:55.934593 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:55.934451 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"75a3f2af-e799-4bfc-bc43-f40ab46c0913","Type":"ContainerDied","Data":"77ba23d9377c25b188c49d964a594a298d1ff3366744dc2b698cc841933ca93d"} Oct 30 10:54:56.082522 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:54:56.082485 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab3b83f_baa8_447b_ad20_76c66c15a7f8.slice/crio-f0c2e73867413e048ce4b8e1e7b351bac988b3766f91636f08321ce4daf83e8c WatchSource:0}: Error finding container f0c2e73867413e048ce4b8e1e7b351bac988b3766f91636f08321ce4daf83e8c: Status 404 returned error can't find the container with id f0c2e73867413e048ce4b8e1e7b351bac988b3766f91636f08321ce4daf83e8c Oct 30 10:54:56.082644 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:56.082595 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 10:54:56.257111 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:54:56.257081 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:54:56.876799 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:56.876720 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8" Oct 30 10:54:56.941501 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:56.941466 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"8ab3b83f-baa8-447b-ad20-76c66c15a7f8","Type":"ContainerStarted","Data":"1e76ebf63bc9ac996b09650186494f379b60a9bd49b5c5697228d6bd15b6b70e"} Oct 30 10:54:56.941667 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:56.941537 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"8ab3b83f-baa8-447b-ad20-76c66c15a7f8","Type":"ContainerStarted","Data":"f0c2e73867413e048ce4b8e1e7b351bac988b3766f91636f08321ce4daf83e8c"} Oct 30 10:54:59.956635 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:59.956554 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"75a3f2af-e799-4bfc-bc43-f40ab46c0913","Type":"ContainerStarted","Data":"f4b8bfdabfb4b4303b7e6e83d20b5006d074b9ea0cc0caa97b8bde7d898677d2"} Oct 30 10:54:59.956635 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:59.956619 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"75a3f2af-e799-4bfc-bc43-f40ab46c0913","Type":"ContainerStarted","Data":"e02a4ab64b7d12c2be335320aed7c7677c3f3914d674e7f7a3790301c6ba5e61"} Oct 30 10:54:59.957026 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:54:59.956824 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:55:00.027545 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:00.027496 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=2.969451877 podStartE2EDuration="12.027482164s" totalImagesPullingTime="5.772519125s" totalInitContainerRuntime="3.285511162s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:54:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:55:00.026726706 +0000 UTC m=+590.528106192" watchObservedRunningTime="2026-10-30 10:55:00.027482164 +0000 UTC m=+590.528861651" Oct 30 10:55:00.962297 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:00.962266 4289 generic.go:356] "Generic (PLEG): container finished" podID="8ab3b83f-baa8-447b-ad20-76c66c15a7f8" containerID="1e76ebf63bc9ac996b09650186494f379b60a9bd49b5c5697228d6bd15b6b70e" exitCode=0 Oct 30 10:55:00.962728 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:00.962344 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"8ab3b83f-baa8-447b-ad20-76c66c15a7f8","Type":"ContainerDied","Data":"1e76ebf63bc9ac996b09650186494f379b60a9bd49b5c5697228d6bd15b6b70e"} Oct 30 10:55:03.981094 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:03.981051 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"8ab3b83f-baa8-447b-ad20-76c66c15a7f8","Type":"ContainerStarted","Data":"13d58b742224e5fae291138650bc03a4f617f437fb6e647d9f7961d144fcbe13"} Oct 30 10:55:04.989023 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:04.988979 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"8ab3b83f-baa8-447b-ad20-76c66c15a7f8","Type":"ContainerStarted","Data":"a4ef55cd9710a012596efb4e6c0c877bc8a32b41f27b98eb4393aa79e9593098"} Oct 30 10:55:05.965314 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:05.965289 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 10:55:05.996356 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:05.996325 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"8ab3b83f-baa8-447b-ad20-76c66c15a7f8","Type":"ContainerStarted","Data":"895d89ae220db3342f65c5ca22e70050b02d4140d094fccde9ed4235a7383c32"} Oct 30 10:55:06.077747 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:06.077691 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=9.734883336 podStartE2EDuration="18.077675431s" totalImagesPullingTime="4.458881413s" totalInitContainerRuntime="3.883910682s" isStatefulPod=true podCreationTimestamp="2026-10-30 10:54:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:55:06.074501642 +0000 UTC m=+596.575881128" watchObservedRunningTime="2026-10-30 10:55:06.077675431 +0000 UTC m=+596.579054917" Oct 30 10:55:10.766867 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:10.766828 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:55:10.767248 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:10.766896 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:55:10.772142 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:10.772120 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:55:11.021827 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:11.021758 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 10:55:18.358004 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:18.357965 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:55:19.987472 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:19.987412 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:55:20.092339 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:20.092306 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:55:20.092558 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:20.092519 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="ca2cf81c-da8d-4ac7-b560-4e264cd006bc" containerName="otc-container" containerID="cri-o://f70f1adfb3b8a68ae73d90425219835abd6e26f8c4814e89caeaacd4eeefc735" gracePeriod=30 Oct 30 10:55:20.556980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:20.556951 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:55:20.736969 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:20.736923 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:55:21.336169 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:21.336145 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:55:21.418768 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:21.418677 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-monitoring-tls-certs\") pod \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\" (UID: \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\") " Oct 30 10:55:21.418924 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:21.418812 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-otc-internal\") pod \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\" (UID: \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\") " Oct 30 10:55:21.418924 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:21.418848 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-kube-api-access-jm49g\" (UniqueName: \"kubernetes.io/projected/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-kube-api-access-jm49g\") pod \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\" (UID: \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\") " Oct 30 10:55:21.418924 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:21.418898 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-tls-certs\") pod \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\" (UID: \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\") " Oct 30 10:55:21.419078 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:21.418933 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-data-science-collector-ta-client-cert\") pod \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\" (UID: \"ca2cf81c-da8d-4ac7-b560-4e264cd006bc\") " Oct 30 10:55:21.419257 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:21.419232 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-otc-internal" pod "ca2cf81c-da8d-4ac7-b560-4e264cd006bc" (UID: "ca2cf81c-da8d-4ac7-b560-4e264cd006bc"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:55:21.421020 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:21.420989 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-data-science-collector-ta-client-cert" pod "ca2cf81c-da8d-4ac7-b560-4e264cd006bc" (UID: "ca2cf81c-da8d-4ac7-b560-4e264cd006bc"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:55:21.421111 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:21.421021 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-tls-certs" pod "ca2cf81c-da8d-4ac7-b560-4e264cd006bc" (UID: "ca2cf81c-da8d-4ac7-b560-4e264cd006bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:55:21.421188 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:21.421172 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-monitoring-tls-certs" pod "ca2cf81c-da8d-4ac7-b560-4e264cd006bc" (UID: "ca2cf81c-da8d-4ac7-b560-4e264cd006bc"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:55:21.421256 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:21.421238 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-kube-api-access-jm49g" pod "ca2cf81c-da8d-4ac7-b560-4e264cd006bc" (UID: "ca2cf81c-da8d-4ac7-b560-4e264cd006bc"). InnerVolumeSpecName "kube-api-access-jm49g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:55:21.520593 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:21.520557 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:55:21.520593 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:21.520586 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm49g\" (UniqueName: \"kubernetes.io/projected/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-kube-api-access-jm49g\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:55:21.520593 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:21.520601 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:55:21.520836 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:21.520614 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:55:21.520836 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:21.520627 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca2cf81c-da8d-4ac7-b560-4e264cd006bc-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:55:22.067050 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.067016 4289 generic.go:356] "Generic (PLEG): container finished" podID="ca2cf81c-da8d-4ac7-b560-4e264cd006bc" containerID="f70f1adfb3b8a68ae73d90425219835abd6e26f8c4814e89caeaacd4eeefc735" exitCode=0 Oct 30 10:55:22.067239 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.067066 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:55:22.067239 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.067094 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ca2cf81c-da8d-4ac7-b560-4e264cd006bc","Type":"ContainerDied","Data":"f70f1adfb3b8a68ae73d90425219835abd6e26f8c4814e89caeaacd4eeefc735"} Oct 30 10:55:22.067239 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.067147 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ca2cf81c-da8d-4ac7-b560-4e264cd006bc","Type":"ContainerDied","Data":"0c3219f3a26a33dde9268f0050923461392bd2d94e3e9fd047b89fd1fdbfcb90"} Oct 30 10:55:22.067239 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.067169 4289 scope.go:122] "RemoveContainer" containerID="f70f1adfb3b8a68ae73d90425219835abd6e26f8c4814e89caeaacd4eeefc735" Oct 30 10:55:22.075269 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.075253 4289 scope.go:122] "RemoveContainer" containerID="f70f1adfb3b8a68ae73d90425219835abd6e26f8c4814e89caeaacd4eeefc735" Oct 30 10:55:22.075578 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:55:22.075554 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f70f1adfb3b8a68ae73d90425219835abd6e26f8c4814e89caeaacd4eeefc735\": container with ID starting with f70f1adfb3b8a68ae73d90425219835abd6e26f8c4814e89caeaacd4eeefc735 not found: ID does not exist" containerID="f70f1adfb3b8a68ae73d90425219835abd6e26f8c4814e89caeaacd4eeefc735" Oct 30 10:55:22.075647 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.075591 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f70f1adfb3b8a68ae73d90425219835abd6e26f8c4814e89caeaacd4eeefc735"} err="failed to get container status \"f70f1adfb3b8a68ae73d90425219835abd6e26f8c4814e89caeaacd4eeefc735\": rpc error: code = NotFound desc = could not find container \"f70f1adfb3b8a68ae73d90425219835abd6e26f8c4814e89caeaacd4eeefc735\": container with ID starting with f70f1adfb3b8a68ae73d90425219835abd6e26f8c4814e89caeaacd4eeefc735 not found: ID does not exist" Oct 30 10:55:22.116366 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.116334 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:55:22.128216 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.128185 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:55:22.163603 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.163570 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:55:22.163996 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.163978 4289 cpu_manager.go:414] "containerMap: removing container" podUID="ca2cf81c-da8d-4ac7-b560-4e264cd006bc" containerName="otc-container" Oct 30 10:55:22.163996 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.163993 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca2cf81c-da8d-4ac7-b560-4e264cd006bc" containerName="otc-container" Oct 30 10:55:22.164140 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.164061 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca2cf81c-da8d-4ac7-b560-4e264cd006bc" containerName="otc-container" Oct 30 10:55:22.168019 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.167998 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:55:22.187111 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.187083 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 10:55:22.187240 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.187154 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 10:55:22.187318 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.187277 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8vrjb\"" Oct 30 10:55:22.188081 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.188040 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 10:55:22.189366 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.189346 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:55:22.190135 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.190118 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-e4cadd38\"" Oct 30 10:55:22.228132 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.228093 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5snkp\" (UniqueName: \"kubernetes.io/projected/abb7e846-e892-44f0-b274-5da28e5a2ef8-kube-api-access-5snkp\") pod \"data-science-collector-collector-1\" (UID: \"abb7e846-e892-44f0-b274-5da28e5a2ef8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:55:22.228278 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.228152 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/abb7e846-e892-44f0-b274-5da28e5a2ef8-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"abb7e846-e892-44f0-b274-5da28e5a2ef8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:55:22.228278 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.228222 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/abb7e846-e892-44f0-b274-5da28e5a2ef8-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"abb7e846-e892-44f0-b274-5da28e5a2ef8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:55:22.228278 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.228257 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abb7e846-e892-44f0-b274-5da28e5a2ef8-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"abb7e846-e892-44f0-b274-5da28e5a2ef8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:55:22.228469 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.228353 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/abb7e846-e892-44f0-b274-5da28e5a2ef8-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"abb7e846-e892-44f0-b274-5da28e5a2ef8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:55:22.329687 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.329596 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5snkp\" (UniqueName: \"kubernetes.io/projected/abb7e846-e892-44f0-b274-5da28e5a2ef8-kube-api-access-5snkp\") pod \"data-science-collector-collector-1\" (UID: \"abb7e846-e892-44f0-b274-5da28e5a2ef8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:55:22.329687 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.329638 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/abb7e846-e892-44f0-b274-5da28e5a2ef8-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"abb7e846-e892-44f0-b274-5da28e5a2ef8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:55:22.329687 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.329675 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/abb7e846-e892-44f0-b274-5da28e5a2ef8-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"abb7e846-e892-44f0-b274-5da28e5a2ef8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:55:22.329954 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.329697 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abb7e846-e892-44f0-b274-5da28e5a2ef8-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"abb7e846-e892-44f0-b274-5da28e5a2ef8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:55:22.329954 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.329760 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/abb7e846-e892-44f0-b274-5da28e5a2ef8-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"abb7e846-e892-44f0-b274-5da28e5a2ef8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:55:22.330495 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.330473 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/abb7e846-e892-44f0-b274-5da28e5a2ef8-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"abb7e846-e892-44f0-b274-5da28e5a2ef8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:55:22.332179 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.332152 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/abb7e846-e892-44f0-b274-5da28e5a2ef8-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"abb7e846-e892-44f0-b274-5da28e5a2ef8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:55:22.332276 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.332194 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abb7e846-e892-44f0-b274-5da28e5a2ef8-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"abb7e846-e892-44f0-b274-5da28e5a2ef8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:55:22.332276 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.332238 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/abb7e846-e892-44f0-b274-5da28e5a2ef8-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"abb7e846-e892-44f0-b274-5da28e5a2ef8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:55:22.352062 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.352030 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5snkp\" (UniqueName: \"kubernetes.io/projected/abb7e846-e892-44f0-b274-5da28e5a2ef8-kube-api-access-5snkp\") pod \"data-science-collector-collector-1\" (UID: \"abb7e846-e892-44f0-b274-5da28e5a2ef8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:55:22.480946 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.480914 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:55:22.773849 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.773816 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:55:22.774387 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:55:22.774357 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb7e846_e892_44f0_b274_5da28e5a2ef8.slice/crio-44e7c3b3e8dfa8634357c54f9b8fb71fc6a3269fdb8cef43ff11cd4a2fa179fa WatchSource:0}: Error finding container 44e7c3b3e8dfa8634357c54f9b8fb71fc6a3269fdb8cef43ff11cd4a2fa179fa: Status 404 returned error can't find the container with id 44e7c3b3e8dfa8634357c54f9b8fb71fc6a3269fdb8cef43ff11cd4a2fa179fa Oct 30 10:55:22.776647 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.776616 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:55:22.776733 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:22.776674 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:55:23.073464 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:23.073361 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"abb7e846-e892-44f0-b274-5da28e5a2ef8","Type":"ContainerStarted","Data":"41406c92afcf484495fea157ad5f3f55c389cd66728b5e39ff338c9140bb4cfc"} Oct 30 10:55:23.073464 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:23.073421 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"abb7e846-e892-44f0-b274-5da28e5a2ef8","Type":"ContainerStarted","Data":"44e7c3b3e8dfa8634357c54f9b8fb71fc6a3269fdb8cef43ff11cd4a2fa179fa"} Oct 30 10:55:23.116193 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:23.116145 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.116132004 podStartE2EDuration="1.116132004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:55:23.114153832 +0000 UTC m=+613.615533316" watchObservedRunningTime="2026-10-30 10:55:23.116132004 +0000 UTC m=+613.617511488" Oct 30 10:55:23.625492 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:23.625455 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca2cf81c-da8d-4ac7-b560-4e264cd006bc" path="/var/lib/kubelet/pods/ca2cf81c-da8d-4ac7-b560-4e264cd006bc/volumes" Oct 30 10:55:26.257009 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:55:26.256972 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:55:37.142694 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:37.142660 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:55:54.556251 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:55:54.556213 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:55:56.257032 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:55:56.256995 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:56:26.257918 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:56:26.257838 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:56:31.482685 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:31.482648 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:56:31.649869 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:31.649833 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:56:45.141301 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:45.141268 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:56:45.306947 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:45.306899 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:56:46.335452 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:46.335397 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:56:46.538608 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:46.538569 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:56:46.651126 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:46.651049 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf"] Oct 30 10:56:46.653836 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:46.653816 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf" Oct 30 10:56:46.672012 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:46.671991 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf"] Oct 30 10:56:46.686337 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:46.686312 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4abf3220-8342-4ca1-896d-7e39e76997c8-tls\") pod \"korrel8r-66d6f7f696-mz6nf\" (UID: \"4abf3220-8342-4ca1-896d-7e39e76997c8\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf" Oct 30 10:56:46.686454 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:46.686350 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4abf3220-8342-4ca1-896d-7e39e76997c8-config\") pod \"korrel8r-66d6f7f696-mz6nf\" (UID: \"4abf3220-8342-4ca1-896d-7e39e76997c8\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf" Oct 30 10:56:46.686503 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:46.686486 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj7gb\" (UniqueName: \"kubernetes.io/projected/4abf3220-8342-4ca1-896d-7e39e76997c8-kube-api-access-vj7gb\") pod \"korrel8r-66d6f7f696-mz6nf\" (UID: \"4abf3220-8342-4ca1-896d-7e39e76997c8\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf" Oct 30 10:56:46.701336 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:46.701308 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:56:46.787173 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:46.787143 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj7gb\" (UniqueName: \"kubernetes.io/projected/4abf3220-8342-4ca1-896d-7e39e76997c8-kube-api-access-vj7gb\") pod \"korrel8r-66d6f7f696-mz6nf\" (UID: \"4abf3220-8342-4ca1-896d-7e39e76997c8\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf" Oct 30 10:56:46.787310 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:46.787208 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4abf3220-8342-4ca1-896d-7e39e76997c8-tls\") pod \"korrel8r-66d6f7f696-mz6nf\" (UID: \"4abf3220-8342-4ca1-896d-7e39e76997c8\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf" Oct 30 10:56:46.787310 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:46.787244 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4abf3220-8342-4ca1-896d-7e39e76997c8-config\") pod \"korrel8r-66d6f7f696-mz6nf\" (UID: \"4abf3220-8342-4ca1-896d-7e39e76997c8\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf" Oct 30 10:56:46.788062 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:46.788037 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4abf3220-8342-4ca1-896d-7e39e76997c8-config\") pod \"korrel8r-66d6f7f696-mz6nf\" (UID: \"4abf3220-8342-4ca1-896d-7e39e76997c8\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf" Oct 30 10:56:46.789551 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:46.789533 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4abf3220-8342-4ca1-896d-7e39e76997c8-tls\") pod \"korrel8r-66d6f7f696-mz6nf\" (UID: \"4abf3220-8342-4ca1-896d-7e39e76997c8\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf" Oct 30 10:56:46.809641 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:46.809620 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj7gb\" (UniqueName: \"kubernetes.io/projected/4abf3220-8342-4ca1-896d-7e39e76997c8-kube-api-access-vj7gb\") pod \"korrel8r-66d6f7f696-mz6nf\" (UID: \"4abf3220-8342-4ca1-896d-7e39e76997c8\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf" Oct 30 10:56:46.965993 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:46.965911 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf" Oct 30 10:56:47.261848 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:56:47.261821 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4abf3220_8342_4ca1_896d_7e39e76997c8.slice/crio-708335b5ae7df61c02c17621a96fb3a7918bfb2193f3f4c2c716df96f60dbaac WatchSource:0}: Error finding container 708335b5ae7df61c02c17621a96fb3a7918bfb2193f3f4c2c716df96f60dbaac: Status 404 returned error can't find the container with id 708335b5ae7df61c02c17621a96fb3a7918bfb2193f3f4c2c716df96f60dbaac Oct 30 10:56:47.262178 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:47.262156 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf"] Oct 30 10:56:47.445106 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:47.445072 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf" event={"ID":"4abf3220-8342-4ca1-896d-7e39e76997c8","Type":"ContainerStarted","Data":"6d71eb9914f7bc563ad21a6e244c0dc6a978f95a3e54514a1c20a76a3e579eea"} Oct 30 10:56:47.445562 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:47.445118 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf" event={"ID":"4abf3220-8342-4ca1-896d-7e39e76997c8","Type":"ContainerStarted","Data":"708335b5ae7df61c02c17621a96fb3a7918bfb2193f3f4c2c716df96f60dbaac"} Oct 30 10:56:47.445562 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:47.445196 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf" Oct 30 10:56:47.487405 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:47.487354 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf" podStartSLOduration=1.487340433 podStartE2EDuration="1.487340433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:56:47.486108776 +0000 UTC m=+697.987488274" watchObservedRunningTime="2026-10-30 10:56:47.487340433 +0000 UTC m=+697.988719955" Oct 30 10:56:48.215149 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:48.215116 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:56:48.215380 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:48.215336 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="abb7e846-e892-44f0-b274-5da28e5a2ef8" containerName="otc-container" containerID="cri-o://41406c92afcf484495fea157ad5f3f55c389cd66728b5e39ff338c9140bb4cfc" gracePeriod=30 Oct 30 10:56:48.392156 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:48.392119 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:56:48.603242 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:48.603206 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:56:49.401469 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:49.401414 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:56:49.613969 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:49.613935 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:56:50.461076 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:50.461041 4289 generic.go:356] "Generic (PLEG): container finished" podID="abb7e846-e892-44f0-b274-5da28e5a2ef8" containerID="41406c92afcf484495fea157ad5f3f55c389cd66728b5e39ff338c9140bb4cfc" exitCode=0 Oct 30 10:56:50.461244 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:50.461120 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"abb7e846-e892-44f0-b274-5da28e5a2ef8","Type":"ContainerDied","Data":"41406c92afcf484495fea157ad5f3f55c389cd66728b5e39ff338c9140bb4cfc"} Oct 30 10:56:50.561017 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:50.560995 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:50.618858 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:50.618781 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abb7e846-e892-44f0-b274-5da28e5a2ef8-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/abb7e846-e892-44f0-b274-5da28e5a2ef8-monitoring-tls-certs\") pod \"abb7e846-e892-44f0-b274-5da28e5a2ef8\" (UID: \"abb7e846-e892-44f0-b274-5da28e5a2ef8\") " Oct 30 10:56:50.618858 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:50.618815 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abb7e846-e892-44f0-b274-5da28e5a2ef8-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/abb7e846-e892-44f0-b274-5da28e5a2ef8-data-science-collector-ta-client-cert\") pod \"abb7e846-e892-44f0-b274-5da28e5a2ef8\" (UID: \"abb7e846-e892-44f0-b274-5da28e5a2ef8\") " Oct 30 10:56:50.618858 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:50.618835 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abb7e846-e892-44f0-b274-5da28e5a2ef8-tls-certs\" (UniqueName: \"kubernetes.io/secret/abb7e846-e892-44f0-b274-5da28e5a2ef8-tls-certs\") pod \"abb7e846-e892-44f0-b274-5da28e5a2ef8\" (UID: \"abb7e846-e892-44f0-b274-5da28e5a2ef8\") " Oct 30 10:56:50.619403 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:50.618873 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abb7e846-e892-44f0-b274-5da28e5a2ef8-otc-internal\" (UniqueName: \"kubernetes.io/configmap/abb7e846-e892-44f0-b274-5da28e5a2ef8-otc-internal\") pod \"abb7e846-e892-44f0-b274-5da28e5a2ef8\" (UID: \"abb7e846-e892-44f0-b274-5da28e5a2ef8\") " Oct 30 10:56:50.619403 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:50.618925 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abb7e846-e892-44f0-b274-5da28e5a2ef8-kube-api-access-5snkp\" (UniqueName: \"kubernetes.io/projected/abb7e846-e892-44f0-b274-5da28e5a2ef8-kube-api-access-5snkp\") pod \"abb7e846-e892-44f0-b274-5da28e5a2ef8\" (UID: \"abb7e846-e892-44f0-b274-5da28e5a2ef8\") " Oct 30 10:56:50.619403 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:50.619290 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abb7e846-e892-44f0-b274-5da28e5a2ef8-otc-internal" pod "abb7e846-e892-44f0-b274-5da28e5a2ef8" (UID: "abb7e846-e892-44f0-b274-5da28e5a2ef8"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:56:50.621036 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:50.621010 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abb7e846-e892-44f0-b274-5da28e5a2ef8-monitoring-tls-certs" pod "abb7e846-e892-44f0-b274-5da28e5a2ef8" (UID: "abb7e846-e892-44f0-b274-5da28e5a2ef8"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:56:50.621164 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:50.621027 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abb7e846-e892-44f0-b274-5da28e5a2ef8-tls-certs" pod "abb7e846-e892-44f0-b274-5da28e5a2ef8" (UID: "abb7e846-e892-44f0-b274-5da28e5a2ef8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:56:50.621376 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:50.621353 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abb7e846-e892-44f0-b274-5da28e5a2ef8-data-science-collector-ta-client-cert" pod "abb7e846-e892-44f0-b274-5da28e5a2ef8" (UID: "abb7e846-e892-44f0-b274-5da28e5a2ef8"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:56:50.621491 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:50.621391 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb7e846-e892-44f0-b274-5da28e5a2ef8-kube-api-access-5snkp" pod "abb7e846-e892-44f0-b274-5da28e5a2ef8" (UID: "abb7e846-e892-44f0-b274-5da28e5a2ef8"). InnerVolumeSpecName "kube-api-access-5snkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:56:50.720411 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:50.720375 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/abb7e846-e892-44f0-b274-5da28e5a2ef8-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:56:50.720411 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:50.720403 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5snkp\" (UniqueName: \"kubernetes.io/projected/abb7e846-e892-44f0-b274-5da28e5a2ef8-kube-api-access-5snkp\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:56:50.720411 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:50.720412 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/abb7e846-e892-44f0-b274-5da28e5a2ef8-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:56:50.720651 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:50.720422 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/abb7e846-e892-44f0-b274-5da28e5a2ef8-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:56:50.720651 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:50.720454 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abb7e846-e892-44f0-b274-5da28e5a2ef8-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:56:51.467299 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.467263 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"abb7e846-e892-44f0-b274-5da28e5a2ef8","Type":"ContainerDied","Data":"44e7c3b3e8dfa8634357c54f9b8fb71fc6a3269fdb8cef43ff11cd4a2fa179fa"} Oct 30 10:56:51.467299 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.467294 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:51.467548 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.467337 4289 scope.go:122] "RemoveContainer" containerID="41406c92afcf484495fea157ad5f3f55c389cd66728b5e39ff338c9140bb4cfc" Oct 30 10:56:51.516890 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.516862 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:56:51.530829 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.530795 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:56:51.557724 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.557694 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:56:51.558188 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.558170 4289 cpu_manager.go:414] "containerMap: removing container" podUID="abb7e846-e892-44f0-b274-5da28e5a2ef8" containerName="otc-container" Oct 30 10:56:51.558276 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.558191 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abb7e846-e892-44f0-b274-5da28e5a2ef8" containerName="otc-container" Oct 30 10:56:51.558332 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.558300 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="abb7e846-e892-44f0-b274-5da28e5a2ef8" containerName="otc-container" Oct 30 10:56:51.561715 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.561694 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:51.578995 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.578969 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-7c92b5b1\"" Oct 30 10:56:51.578995 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.578987 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 10:56:51.579172 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.579151 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 10:56:51.579249 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.579236 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 10:56:51.579299 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.579288 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8vrjb\"" Oct 30 10:56:51.581161 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.581137 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:56:51.624479 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.624453 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abb7e846-e892-44f0-b274-5da28e5a2ef8" path="/var/lib/kubelet/pods/abb7e846-e892-44f0-b274-5da28e5a2ef8/volumes" Oct 30 10:56:51.627650 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.627628 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:51.627751 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.627684 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:51.627839 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.627816 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c98kb\" (UniqueName: \"kubernetes.io/projected/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-kube-api-access-c98kb\") pod \"data-science-collector-collector-1\" (UID: \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:51.627899 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.627864 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:51.627947 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.627914 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:51.728997 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.728965 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:51.729158 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.729008 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c98kb\" (UniqueName: \"kubernetes.io/projected/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-kube-api-access-c98kb\") pod \"data-science-collector-collector-1\" (UID: \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:51.729158 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.729044 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:51.729158 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.729069 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:51.729158 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.729139 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:51.729813 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.729790 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:51.731408 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.731381 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:51.731516 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.731407 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:51.731516 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.731495 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:51.751447 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.751406 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c98kb\" (UniqueName: \"kubernetes.io/projected/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-kube-api-access-c98kb\") pod \"data-science-collector-collector-1\" (UID: \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:51.873712 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:51.873680 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:52.166913 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:52.166891 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:56:52.167198 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:56:52.167167 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6d01d2_cd5f_4121_9fea_0b59c3c18ed6.slice/crio-5f39ce590a420f50fcb9e5de9bfd0eb7e52828b4e02ea56ce4a493121f8f9891 WatchSource:0}: Error finding container 5f39ce590a420f50fcb9e5de9bfd0eb7e52828b4e02ea56ce4a493121f8f9891: Status 404 returned error can't find the container with id 5f39ce590a420f50fcb9e5de9bfd0eb7e52828b4e02ea56ce4a493121f8f9891 Oct 30 10:56:52.169441 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:52.169392 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:56:52.169561 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:52.169481 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:56:52.473958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:52.473919 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6","Type":"ContainerStarted","Data":"da59391150f6702665539298d5e8849e47565951887b31731fe831fdc00c70d4"} Oct 30 10:56:52.474127 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:52.473982 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6","Type":"ContainerStarted","Data":"5f39ce590a420f50fcb9e5de9bfd0eb7e52828b4e02ea56ce4a493121f8f9891"} Oct 30 10:56:52.514198 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:52.514140 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.514123272 podStartE2EDuration="1.514123272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:56:52.514008135 +0000 UTC m=+703.015387620" watchObservedRunningTime="2026-10-30 10:56:52.514123272 +0000 UTC m=+703.015502760" Oct 30 10:56:54.665694 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:54.665661 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:56:54.968776 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:54.968694 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:56:55.146645 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:55.146586 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:56:55.306525 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:55.306485 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf"] Oct 30 10:56:55.306744 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:55.306718 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf" podUID="4abf3220-8342-4ca1-896d-7e39e76997c8" containerName="korrel8r" containerID="cri-o://6d71eb9914f7bc563ad21a6e244c0dc6a978f95a3e54514a1c20a76a3e579eea" gracePeriod=30 Oct 30 10:56:55.308605 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:55.308586 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf" Oct 30 10:56:55.316249 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:55.316216 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:56:55.495299 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:55.495262 4289 generic.go:356] "Generic (PLEG): container finished" podID="4abf3220-8342-4ca1-896d-7e39e76997c8" containerID="6d71eb9914f7bc563ad21a6e244c0dc6a978f95a3e54514a1c20a76a3e579eea" exitCode=0 Oct 30 10:56:55.495416 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:55.495306 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf" event={"ID":"4abf3220-8342-4ca1-896d-7e39e76997c8","Type":"ContainerDied","Data":"6d71eb9914f7bc563ad21a6e244c0dc6a978f95a3e54514a1c20a76a3e579eea"} Oct 30 10:56:55.560755 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:55.560697 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf" Oct 30 10:56:55.665868 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:55.665839 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4abf3220-8342-4ca1-896d-7e39e76997c8-tls\" (UniqueName: \"kubernetes.io/secret/4abf3220-8342-4ca1-896d-7e39e76997c8-tls\") pod \"4abf3220-8342-4ca1-896d-7e39e76997c8\" (UID: \"4abf3220-8342-4ca1-896d-7e39e76997c8\") " Oct 30 10:56:55.666233 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:55.665930 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4abf3220-8342-4ca1-896d-7e39e76997c8-kube-api-access-vj7gb\" (UniqueName: \"kubernetes.io/projected/4abf3220-8342-4ca1-896d-7e39e76997c8-kube-api-access-vj7gb\") pod \"4abf3220-8342-4ca1-896d-7e39e76997c8\" (UID: \"4abf3220-8342-4ca1-896d-7e39e76997c8\") " Oct 30 10:56:55.666233 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:55.666007 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4abf3220-8342-4ca1-896d-7e39e76997c8-config\" (UniqueName: \"kubernetes.io/configmap/4abf3220-8342-4ca1-896d-7e39e76997c8-config\") pod \"4abf3220-8342-4ca1-896d-7e39e76997c8\" (UID: \"4abf3220-8342-4ca1-896d-7e39e76997c8\") " Oct 30 10:56:55.666425 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:55.666399 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4abf3220-8342-4ca1-896d-7e39e76997c8-config" pod "4abf3220-8342-4ca1-896d-7e39e76997c8" (UID: "4abf3220-8342-4ca1-896d-7e39e76997c8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:56:55.668019 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:55.667991 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4abf3220-8342-4ca1-896d-7e39e76997c8-tls" pod "4abf3220-8342-4ca1-896d-7e39e76997c8" (UID: "4abf3220-8342-4ca1-896d-7e39e76997c8"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:56:55.668113 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:55.668058 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4abf3220-8342-4ca1-896d-7e39e76997c8-kube-api-access-vj7gb" pod "4abf3220-8342-4ca1-896d-7e39e76997c8" (UID: "4abf3220-8342-4ca1-896d-7e39e76997c8"). InnerVolumeSpecName "kube-api-access-vj7gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:56:55.767159 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:55.767125 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4abf3220-8342-4ca1-896d-7e39e76997c8-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:56:55.767159 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:55.767161 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj7gb\" (UniqueName: \"kubernetes.io/projected/4abf3220-8342-4ca1-896d-7e39e76997c8-kube-api-access-vj7gb\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:56:55.767344 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:55.767179 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4abf3220-8342-4ca1-896d-7e39e76997c8-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:56:56.257677 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:56:56.257641 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:56:56.501014 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:56.500979 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf" event={"ID":"4abf3220-8342-4ca1-896d-7e39e76997c8","Type":"ContainerDied","Data":"708335b5ae7df61c02c17621a96fb3a7918bfb2193f3f4c2c716df96f60dbaac"} Oct 30 10:56:56.501014 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:56.501005 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf" Oct 30 10:56:56.501285 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:56.501036 4289 scope.go:122] "RemoveContainer" containerID="6d71eb9914f7bc563ad21a6e244c0dc6a978f95a3e54514a1c20a76a3e579eea" Oct 30 10:56:56.557668 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:56.552257 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf"] Oct 30 10:56:56.562489 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:56.562467 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-mz6nf"] Oct 30 10:56:57.187027 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.186996 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:56:57.187461 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.187199 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6" containerName="otc-container" containerID="cri-o://da59391150f6702665539298d5e8849e47565951887b31731fe831fdc00c70d4" gracePeriod=30 Oct 30 10:56:57.365469 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.365387 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:56:57.439154 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.439100 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:57.480497 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.480464 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-monitoring-tls-certs\") pod \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\" (UID: \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\") " Oct 30 10:56:57.480683 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.480533 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-otc-internal\" (UniqueName: \"kubernetes.io/configmap/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-otc-internal\") pod \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\" (UID: \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\") " Oct 30 10:56:57.480683 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.480598 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-kube-api-access-c98kb\" (UniqueName: \"kubernetes.io/projected/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-kube-api-access-c98kb\") pod \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\" (UID: \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\") " Oct 30 10:56:57.480683 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.480628 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-data-science-collector-ta-client-cert\") pod \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\" (UID: \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\") " Oct 30 10:56:57.480683 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.480671 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-tls-certs\") pod \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\" (UID: \"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6\") " Oct 30 10:56:57.482360 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.482316 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-otc-internal" pod "7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6" (UID: "7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:56:57.484852 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.484813 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-tls-certs" pod "7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6" (UID: "7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:56:57.484954 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.484873 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-data-science-collector-ta-client-cert" pod "7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6" (UID: "7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:56:57.484954 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.484895 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-monitoring-tls-certs" pod "7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6" (UID: "7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:56:57.485290 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.485259 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-kube-api-access-c98kb" pod "7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6" (UID: "7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6"). InnerVolumeSpecName "kube-api-access-c98kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:56:57.506903 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.506872 4289 generic.go:356] "Generic (PLEG): container finished" podID="7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6" containerID="da59391150f6702665539298d5e8849e47565951887b31731fe831fdc00c70d4" exitCode=0 Oct 30 10:56:57.507023 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.506936 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:57.507023 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.506959 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6","Type":"ContainerDied","Data":"da59391150f6702665539298d5e8849e47565951887b31731fe831fdc00c70d4"} Oct 30 10:56:57.507140 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.507026 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6","Type":"ContainerDied","Data":"5f39ce590a420f50fcb9e5de9bfd0eb7e52828b4e02ea56ce4a493121f8f9891"} Oct 30 10:56:57.507140 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.507056 4289 scope.go:122] "RemoveContainer" containerID="da59391150f6702665539298d5e8849e47565951887b31731fe831fdc00c70d4" Oct 30 10:56:57.515628 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.515604 4289 scope.go:122] "RemoveContainer" containerID="da59391150f6702665539298d5e8849e47565951887b31731fe831fdc00c70d4" Oct 30 10:56:57.515883 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:56:57.515861 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da59391150f6702665539298d5e8849e47565951887b31731fe831fdc00c70d4\": container with ID starting with da59391150f6702665539298d5e8849e47565951887b31731fe831fdc00c70d4 not found: ID does not exist" containerID="da59391150f6702665539298d5e8849e47565951887b31731fe831fdc00c70d4" Oct 30 10:56:57.515939 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.515892 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da59391150f6702665539298d5e8849e47565951887b31731fe831fdc00c70d4"} err="failed to get container status \"da59391150f6702665539298d5e8849e47565951887b31731fe831fdc00c70d4\": rpc error: code = NotFound desc = could not find container \"da59391150f6702665539298d5e8849e47565951887b31731fe831fdc00c70d4\": container with ID starting with da59391150f6702665539298d5e8849e47565951887b31731fe831fdc00c70d4 not found: ID does not exist" Oct 30 10:56:57.528012 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.527990 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:56:57.558679 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.558653 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:56:57.564040 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.564018 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:56:57.581790 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.581768 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:56:57.581790 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.581789 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:56:57.581914 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.581800 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:56:57.581914 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.581811 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c98kb\" (UniqueName: \"kubernetes.io/projected/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-kube-api-access-c98kb\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:56:57.581914 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.581821 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:56:57.599923 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.599899 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:56:57.600320 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.600307 4289 cpu_manager.go:414] "containerMap: removing container" podUID="7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6" containerName="otc-container" Oct 30 10:56:57.600361 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.600322 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6" containerName="otc-container" Oct 30 10:56:57.600361 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.600346 4289 cpu_manager.go:414] "containerMap: removing container" podUID="4abf3220-8342-4ca1-896d-7e39e76997c8" containerName="korrel8r" Oct 30 10:56:57.600361 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.600352 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4abf3220-8342-4ca1-896d-7e39e76997c8" containerName="korrel8r" Oct 30 10:56:57.600491 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.600417 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6" containerName="otc-container" Oct 30 10:56:57.600491 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.600441 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="4abf3220-8342-4ca1-896d-7e39e76997c8" containerName="korrel8r" Oct 30 10:56:57.604862 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.604846 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:57.621384 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.621357 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 10:56:57.621507 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.621452 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 10:56:57.621582 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.621521 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 10:56:57.621582 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.621523 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8vrjb\"" Oct 30 10:56:57.621582 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.621552 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-e4cadd38\"" Oct 30 10:56:57.626109 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.626081 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4abf3220-8342-4ca1-896d-7e39e76997c8" path="/var/lib/kubelet/pods/4abf3220-8342-4ca1-896d-7e39e76997c8/volumes" Oct 30 10:56:57.626683 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.626663 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6" path="/var/lib/kubelet/pods/7a6d01d2-cd5f-4121-9fea-0b59c3c18ed6/volumes" Oct 30 10:56:57.627009 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.626992 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:56:57.683196 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.683164 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19902b06-5c78-4daf-bef0-0369fea98657-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"19902b06-5c78-4daf-bef0-0369fea98657\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:57.683196 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.683196 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4f4s\" (UniqueName: \"kubernetes.io/projected/19902b06-5c78-4daf-bef0-0369fea98657-kube-api-access-z4f4s\") pod \"data-science-collector-collector-1\" (UID: \"19902b06-5c78-4daf-bef0-0369fea98657\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:57.683390 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.683218 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/19902b06-5c78-4daf-bef0-0369fea98657-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"19902b06-5c78-4daf-bef0-0369fea98657\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:57.683390 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.683281 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/19902b06-5c78-4daf-bef0-0369fea98657-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"19902b06-5c78-4daf-bef0-0369fea98657\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:57.683390 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.683333 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/19902b06-5c78-4daf-bef0-0369fea98657-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"19902b06-5c78-4daf-bef0-0369fea98657\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:57.784544 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.784510 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19902b06-5c78-4daf-bef0-0369fea98657-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"19902b06-5c78-4daf-bef0-0369fea98657\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:57.784544 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.784546 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4f4s\" (UniqueName: \"kubernetes.io/projected/19902b06-5c78-4daf-bef0-0369fea98657-kube-api-access-z4f4s\") pod \"data-science-collector-collector-1\" (UID: \"19902b06-5c78-4daf-bef0-0369fea98657\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:57.784759 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.784569 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/19902b06-5c78-4daf-bef0-0369fea98657-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"19902b06-5c78-4daf-bef0-0369fea98657\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:57.784759 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.784592 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/19902b06-5c78-4daf-bef0-0369fea98657-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"19902b06-5c78-4daf-bef0-0369fea98657\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:57.784759 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.784643 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/19902b06-5c78-4daf-bef0-0369fea98657-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"19902b06-5c78-4daf-bef0-0369fea98657\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:57.785333 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.785302 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/19902b06-5c78-4daf-bef0-0369fea98657-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"19902b06-5c78-4daf-bef0-0369fea98657\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:57.787058 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.787030 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/19902b06-5c78-4daf-bef0-0369fea98657-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"19902b06-5c78-4daf-bef0-0369fea98657\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:57.787162 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.787095 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19902b06-5c78-4daf-bef0-0369fea98657-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"19902b06-5c78-4daf-bef0-0369fea98657\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:57.787162 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.787107 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/19902b06-5c78-4daf-bef0-0369fea98657-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"19902b06-5c78-4daf-bef0-0369fea98657\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:57.807337 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.807310 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4f4s\" (UniqueName: \"kubernetes.io/projected/19902b06-5c78-4daf-bef0-0369fea98657-kube-api-access-z4f4s\") pod \"data-science-collector-collector-1\" (UID: \"19902b06-5c78-4daf-bef0-0369fea98657\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:57.917048 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:57.917012 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:56:58.211944 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:58.211906 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:56:58.212792 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:56:58.212761 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19902b06_5c78_4daf_bef0_0369fea98657.slice/crio-652a1c685cf17f96b092bddfe20d7650a6e93c318712cefe80a46d3ce89154a7 WatchSource:0}: Error finding container 652a1c685cf17f96b092bddfe20d7650a6e93c318712cefe80a46d3ce89154a7: Status 404 returned error can't find the container with id 652a1c685cf17f96b092bddfe20d7650a6e93c318712cefe80a46d3ce89154a7 Oct 30 10:56:58.215025 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:58.214991 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:56:58.215127 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:58.215063 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:56:58.260379 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:58.260289 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:56:58.515418 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:58.515387 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"19902b06-5c78-4daf-bef0-0369fea98657","Type":"ContainerStarted","Data":"f0396c3d67f734065ea907819e0f21fde01e24992a313e0fca66215cc623dd06"} Oct 30 10:56:58.515622 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:58.515461 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"19902b06-5c78-4daf-bef0-0369fea98657","Type":"ContainerStarted","Data":"652a1c685cf17f96b092bddfe20d7650a6e93c318712cefe80a46d3ce89154a7"} Oct 30 10:56:58.557909 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:58.557850 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.5578365509999998 podStartE2EDuration="1.557836551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:56:58.556670622 +0000 UTC m=+709.058050107" watchObservedRunningTime="2026-10-30 10:56:58.557836551 +0000 UTC m=+709.059216036" Oct 30 10:56:59.470925 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:56:59.470890 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:04.457064 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:04.457032 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:04.667050 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:04.667011 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:04.745490 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:04.745463 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr"] Oct 30 10:57:04.749035 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:04.749016 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr" Oct 30 10:57:04.766297 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:04.766272 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr"] Oct 30 10:57:04.848552 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:04.848517 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19bc4d6e-0dc6-461a-a5ae-24962bb4231e-config\") pod \"korrel8r-66d6f7f696-jgmlr\" (UID: \"19bc4d6e-0dc6-461a-a5ae-24962bb4231e\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr" Oct 30 10:57:04.848701 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:04.848560 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5mhr\" (UniqueName: \"kubernetes.io/projected/19bc4d6e-0dc6-461a-a5ae-24962bb4231e-kube-api-access-m5mhr\") pod \"korrel8r-66d6f7f696-jgmlr\" (UID: \"19bc4d6e-0dc6-461a-a5ae-24962bb4231e\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr" Oct 30 10:57:04.848767 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:04.848742 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/19bc4d6e-0dc6-461a-a5ae-24962bb4231e-tls\") pod \"korrel8r-66d6f7f696-jgmlr\" (UID: \"19bc4d6e-0dc6-461a-a5ae-24962bb4231e\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr" Oct 30 10:57:04.949202 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:04.949168 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/19bc4d6e-0dc6-461a-a5ae-24962bb4231e-tls\") pod \"korrel8r-66d6f7f696-jgmlr\" (UID: \"19bc4d6e-0dc6-461a-a5ae-24962bb4231e\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr" Oct 30 10:57:04.949360 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:04.949220 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19bc4d6e-0dc6-461a-a5ae-24962bb4231e-config\") pod \"korrel8r-66d6f7f696-jgmlr\" (UID: \"19bc4d6e-0dc6-461a-a5ae-24962bb4231e\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr" Oct 30 10:57:04.949360 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:04.949244 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5mhr\" (UniqueName: \"kubernetes.io/projected/19bc4d6e-0dc6-461a-a5ae-24962bb4231e-kube-api-access-m5mhr\") pod \"korrel8r-66d6f7f696-jgmlr\" (UID: \"19bc4d6e-0dc6-461a-a5ae-24962bb4231e\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr" Oct 30 10:57:04.949965 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:04.949945 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19bc4d6e-0dc6-461a-a5ae-24962bb4231e-config\") pod \"korrel8r-66d6f7f696-jgmlr\" (UID: \"19bc4d6e-0dc6-461a-a5ae-24962bb4231e\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr" Oct 30 10:57:04.951608 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:04.951589 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/19bc4d6e-0dc6-461a-a5ae-24962bb4231e-tls\") pod \"korrel8r-66d6f7f696-jgmlr\" (UID: \"19bc4d6e-0dc6-461a-a5ae-24962bb4231e\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr" Oct 30 10:57:04.971617 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:04.971590 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5mhr\" (UniqueName: \"kubernetes.io/projected/19bc4d6e-0dc6-461a-a5ae-24962bb4231e-kube-api-access-m5mhr\") pod \"korrel8r-66d6f7f696-jgmlr\" (UID: \"19bc4d6e-0dc6-461a-a5ae-24962bb4231e\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr" Oct 30 10:57:05.061977 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:05.061897 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr" Oct 30 10:57:05.361599 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:05.361565 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr"] Oct 30 10:57:05.362155 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:57:05.362131 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19bc4d6e_0dc6_461a_a5ae_24962bb4231e.slice/crio-e659c87dc7e6274320d285b1f8f346182ce573ce43d88aec89e1e6a2f1c68f65 WatchSource:0}: Error finding container e659c87dc7e6274320d285b1f8f346182ce573ce43d88aec89e1e6a2f1c68f65: Status 404 returned error can't find the container with id e659c87dc7e6274320d285b1f8f346182ce573ce43d88aec89e1e6a2f1c68f65 Oct 30 10:57:05.548758 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:05.548728 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr" event={"ID":"19bc4d6e-0dc6-461a-a5ae-24962bb4231e","Type":"ContainerStarted","Data":"771758debfdeff44aa2982abcacdae118af99f714beb6bfa294e8beeaf29649d"} Oct 30 10:57:05.549121 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:05.548779 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr" event={"ID":"19bc4d6e-0dc6-461a-a5ae-24962bb4231e","Type":"ContainerStarted","Data":"e659c87dc7e6274320d285b1f8f346182ce573ce43d88aec89e1e6a2f1c68f65"} Oct 30 10:57:05.549121 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:05.548810 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr" Oct 30 10:57:06.221343 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.221284 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr" podStartSLOduration=2.221270719 podStartE2EDuration="2.221270719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:57:05.587761367 +0000 UTC m=+716.089140856" watchObservedRunningTime="2026-10-30 10:57:06.221270719 +0000 UTC m=+716.722650204" Oct 30 10:57:06.222777 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.222756 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:06.222950 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.222931 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="19902b06-5c78-4daf-bef0-0369fea98657" containerName="otc-container" containerID="cri-o://f0396c3d67f734065ea907819e0f21fde01e24992a313e0fca66215cc623dd06" gracePeriod=30 Oct 30 10:57:06.397848 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.397810 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:06.473922 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.473872 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:06.560264 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.560224 4289 generic.go:356] "Generic (PLEG): container finished" podID="19902b06-5c78-4daf-bef0-0369fea98657" containerID="f0396c3d67f734065ea907819e0f21fde01e24992a313e0fca66215cc623dd06" exitCode=0 Oct 30 10:57:06.560690 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.560288 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:06.560690 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.560300 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"19902b06-5c78-4daf-bef0-0369fea98657","Type":"ContainerDied","Data":"f0396c3d67f734065ea907819e0f21fde01e24992a313e0fca66215cc623dd06"} Oct 30 10:57:06.560690 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.560392 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"19902b06-5c78-4daf-bef0-0369fea98657","Type":"ContainerDied","Data":"652a1c685cf17f96b092bddfe20d7650a6e93c318712cefe80a46d3ce89154a7"} Oct 30 10:57:06.560690 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.560425 4289 scope.go:122] "RemoveContainer" containerID="f0396c3d67f734065ea907819e0f21fde01e24992a313e0fca66215cc623dd06" Oct 30 10:57:06.561020 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.560994 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19902b06-5c78-4daf-bef0-0369fea98657-kube-api-access-z4f4s\" (UniqueName: \"kubernetes.io/projected/19902b06-5c78-4daf-bef0-0369fea98657-kube-api-access-z4f4s\") pod \"19902b06-5c78-4daf-bef0-0369fea98657\" (UID: \"19902b06-5c78-4daf-bef0-0369fea98657\") " Oct 30 10:57:06.561155 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.561035 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19902b06-5c78-4daf-bef0-0369fea98657-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/19902b06-5c78-4daf-bef0-0369fea98657-monitoring-tls-certs\") pod \"19902b06-5c78-4daf-bef0-0369fea98657\" (UID: \"19902b06-5c78-4daf-bef0-0369fea98657\") " Oct 30 10:57:06.561155 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.561076 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19902b06-5c78-4daf-bef0-0369fea98657-tls-certs\" (UniqueName: \"kubernetes.io/secret/19902b06-5c78-4daf-bef0-0369fea98657-tls-certs\") pod \"19902b06-5c78-4daf-bef0-0369fea98657\" (UID: \"19902b06-5c78-4daf-bef0-0369fea98657\") " Oct 30 10:57:06.561155 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.561111 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19902b06-5c78-4daf-bef0-0369fea98657-otc-internal\" (UniqueName: \"kubernetes.io/configmap/19902b06-5c78-4daf-bef0-0369fea98657-otc-internal\") pod \"19902b06-5c78-4daf-bef0-0369fea98657\" (UID: \"19902b06-5c78-4daf-bef0-0369fea98657\") " Oct 30 10:57:06.561155 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.561143 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19902b06-5c78-4daf-bef0-0369fea98657-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/19902b06-5c78-4daf-bef0-0369fea98657-data-science-collector-ta-client-cert\") pod \"19902b06-5c78-4daf-bef0-0369fea98657\" (UID: \"19902b06-5c78-4daf-bef0-0369fea98657\") " Oct 30 10:57:06.561614 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.561580 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19902b06-5c78-4daf-bef0-0369fea98657-otc-internal" pod "19902b06-5c78-4daf-bef0-0369fea98657" (UID: "19902b06-5c78-4daf-bef0-0369fea98657"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:57:06.566832 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.566792 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19902b06-5c78-4daf-bef0-0369fea98657-data-science-collector-ta-client-cert" pod "19902b06-5c78-4daf-bef0-0369fea98657" (UID: "19902b06-5c78-4daf-bef0-0369fea98657"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:57:06.566832 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.566827 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19902b06-5c78-4daf-bef0-0369fea98657-tls-certs" pod "19902b06-5c78-4daf-bef0-0369fea98657" (UID: "19902b06-5c78-4daf-bef0-0369fea98657"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:57:06.566998 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.566869 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19902b06-5c78-4daf-bef0-0369fea98657-monitoring-tls-certs" pod "19902b06-5c78-4daf-bef0-0369fea98657" (UID: "19902b06-5c78-4daf-bef0-0369fea98657"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:57:06.566998 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.566879 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19902b06-5c78-4daf-bef0-0369fea98657-kube-api-access-z4f4s" pod "19902b06-5c78-4daf-bef0-0369fea98657" (UID: "19902b06-5c78-4daf-bef0-0369fea98657"). InnerVolumeSpecName "kube-api-access-z4f4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:57:06.576863 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.576836 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:06.578008 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.577984 4289 scope.go:122] "RemoveContainer" containerID="f0396c3d67f734065ea907819e0f21fde01e24992a313e0fca66215cc623dd06" Oct 30 10:57:06.578289 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:57:06.578262 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0396c3d67f734065ea907819e0f21fde01e24992a313e0fca66215cc623dd06\": container with ID starting with f0396c3d67f734065ea907819e0f21fde01e24992a313e0fca66215cc623dd06 not found: ID does not exist" containerID="f0396c3d67f734065ea907819e0f21fde01e24992a313e0fca66215cc623dd06" Oct 30 10:57:06.578356 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.578295 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0396c3d67f734065ea907819e0f21fde01e24992a313e0fca66215cc623dd06"} err="failed to get container status \"f0396c3d67f734065ea907819e0f21fde01e24992a313e0fca66215cc623dd06\": rpc error: code = NotFound desc = could not find container \"f0396c3d67f734065ea907819e0f21fde01e24992a313e0fca66215cc623dd06\": container with ID starting with f0396c3d67f734065ea907819e0f21fde01e24992a313e0fca66215cc623dd06 not found: ID does not exist" Oct 30 10:57:06.662721 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.662688 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4f4s\" (UniqueName: \"kubernetes.io/projected/19902b06-5c78-4daf-bef0-0369fea98657-kube-api-access-z4f4s\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:06.662721 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.662713 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/19902b06-5c78-4daf-bef0-0369fea98657-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:06.662721 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.662724 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19902b06-5c78-4daf-bef0-0369fea98657-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:06.662946 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.662733 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/19902b06-5c78-4daf-bef0-0369fea98657-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:06.662946 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.662742 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/19902b06-5c78-4daf-bef0-0369fea98657-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:06.903572 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.903540 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:06.915600 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.915570 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:06.954649 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.954624 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:06.955018 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.955003 4289 cpu_manager.go:414] "containerMap: removing container" podUID="19902b06-5c78-4daf-bef0-0369fea98657" containerName="otc-container" Oct 30 10:57:06.955018 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.955017 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19902b06-5c78-4daf-bef0-0369fea98657" containerName="otc-container" Oct 30 10:57:06.955156 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.955099 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="19902b06-5c78-4daf-bef0-0369fea98657" containerName="otc-container" Oct 30 10:57:06.959477 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.959459 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:06.976078 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.976057 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 10:57:06.976078 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.976061 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 10:57:06.976078 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.976058 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8vrjb\"" Oct 30 10:57:06.976369 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.976107 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 10:57:06.976369 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.976230 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-7c92b5b1\"" Oct 30 10:57:06.978390 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:06.978367 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:07.066024 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:07.065987 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec28358-f41b-478f-be0b-b85b3932c116-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6ec28358-f41b-478f-be0b-b85b3932c116\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:07.066184 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:07.066070 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6ec28358-f41b-478f-be0b-b85b3932c116-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"6ec28358-f41b-478f-be0b-b85b3932c116\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:07.066184 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:07.066128 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec28358-f41b-478f-be0b-b85b3932c116-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6ec28358-f41b-478f-be0b-b85b3932c116\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:07.066184 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:07.066152 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6ec28358-f41b-478f-be0b-b85b3932c116-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"6ec28358-f41b-478f-be0b-b85b3932c116\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:07.066331 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:07.066193 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79v42\" (UniqueName: \"kubernetes.io/projected/6ec28358-f41b-478f-be0b-b85b3932c116-kube-api-access-79v42\") pod \"data-science-collector-collector-1\" (UID: \"6ec28358-f41b-478f-be0b-b85b3932c116\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:07.166800 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:07.166709 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec28358-f41b-478f-be0b-b85b3932c116-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6ec28358-f41b-478f-be0b-b85b3932c116\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:07.166800 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:07.166748 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6ec28358-f41b-478f-be0b-b85b3932c116-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"6ec28358-f41b-478f-be0b-b85b3932c116\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:07.166800 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:07.166800 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec28358-f41b-478f-be0b-b85b3932c116-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6ec28358-f41b-478f-be0b-b85b3932c116\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:07.167080 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:07.166822 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6ec28358-f41b-478f-be0b-b85b3932c116-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"6ec28358-f41b-478f-be0b-b85b3932c116\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:07.167080 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:07.166844 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79v42\" (UniqueName: \"kubernetes.io/projected/6ec28358-f41b-478f-be0b-b85b3932c116-kube-api-access-79v42\") pod \"data-science-collector-collector-1\" (UID: \"6ec28358-f41b-478f-be0b-b85b3932c116\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:07.167622 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:07.167582 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6ec28358-f41b-478f-be0b-b85b3932c116-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"6ec28358-f41b-478f-be0b-b85b3932c116\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:07.169438 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:07.169402 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec28358-f41b-478f-be0b-b85b3932c116-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6ec28358-f41b-478f-be0b-b85b3932c116\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:07.169532 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:07.169453 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec28358-f41b-478f-be0b-b85b3932c116-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6ec28358-f41b-478f-be0b-b85b3932c116\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:07.169532 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:07.169522 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6ec28358-f41b-478f-be0b-b85b3932c116-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"6ec28358-f41b-478f-be0b-b85b3932c116\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:07.191255 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:07.191231 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79v42\" (UniqueName: \"kubernetes.io/projected/6ec28358-f41b-478f-be0b-b85b3932c116-kube-api-access-79v42\") pod \"data-science-collector-collector-1\" (UID: \"6ec28358-f41b-478f-be0b-b85b3932c116\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:07.271841 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:07.271810 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:07.437112 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:07.437031 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:07.568634 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:07.568604 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:07.569358 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:57:07.569335 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec28358_f41b_478f_be0b_b85b3932c116.slice/crio-ca5f479385c6dad3c0b7840e26a5d1f307b62c8fa034fdbc5bd75de565a2d172 WatchSource:0}: Error finding container ca5f479385c6dad3c0b7840e26a5d1f307b62c8fa034fdbc5bd75de565a2d172: Status 404 returned error can't find the container with id ca5f479385c6dad3c0b7840e26a5d1f307b62c8fa034fdbc5bd75de565a2d172 Oct 30 10:57:07.571724 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:07.571692 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:57:07.571794 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:07.571751 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:57:07.617126 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:07.617102 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:07.625472 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:07.625445 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19902b06-5c78-4daf-bef0-0369fea98657" path="/var/lib/kubelet/pods/19902b06-5c78-4daf-bef0-0369fea98657/volumes" Oct 30 10:57:08.572274 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:08.572236 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6ec28358-f41b-478f-be0b-b85b3932c116","Type":"ContainerStarted","Data":"7dad3636084297a24d0f46fd45718058039dc3646fdb92e46c36a37e36dda4e0"} Oct 30 10:57:08.572778 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:08.572292 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6ec28358-f41b-478f-be0b-b85b3932c116","Type":"ContainerStarted","Data":"ca5f479385c6dad3c0b7840e26a5d1f307b62c8fa034fdbc5bd75de565a2d172"} Oct 30 10:57:08.616879 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:08.616828 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.6168156529999997 podStartE2EDuration="2.616815653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:57:08.616759357 +0000 UTC m=+719.118138841" watchObservedRunningTime="2026-10-30 10:57:08.616815653 +0000 UTC m=+719.118195138" Oct 30 10:57:12.850389 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:12.850339 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:13.042839 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:13.042800 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:13.171570 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:13.171486 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr"] Oct 30 10:57:13.171708 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:13.171672 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr" podUID="19bc4d6e-0dc6-461a-a5ae-24962bb4231e" containerName="korrel8r" containerID="cri-o://771758debfdeff44aa2982abcacdae118af99f714beb6bfa294e8beeaf29649d" gracePeriod=30 Oct 30 10:57:13.173597 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:13.173575 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr" Oct 30 10:57:13.211641 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:13.211604 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:13.416676 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:13.416653 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr" Oct 30 10:57:13.523909 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:13.523880 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19bc4d6e-0dc6-461a-a5ae-24962bb4231e-tls\" (UniqueName: \"kubernetes.io/secret/19bc4d6e-0dc6-461a-a5ae-24962bb4231e-tls\") pod \"19bc4d6e-0dc6-461a-a5ae-24962bb4231e\" (UID: \"19bc4d6e-0dc6-461a-a5ae-24962bb4231e\") " Oct 30 10:57:13.524091 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:13.523929 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19bc4d6e-0dc6-461a-a5ae-24962bb4231e-kube-api-access-m5mhr\" (UniqueName: \"kubernetes.io/projected/19bc4d6e-0dc6-461a-a5ae-24962bb4231e-kube-api-access-m5mhr\") pod \"19bc4d6e-0dc6-461a-a5ae-24962bb4231e\" (UID: \"19bc4d6e-0dc6-461a-a5ae-24962bb4231e\") " Oct 30 10:57:13.524091 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:13.524011 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19bc4d6e-0dc6-461a-a5ae-24962bb4231e-config\" (UniqueName: \"kubernetes.io/configmap/19bc4d6e-0dc6-461a-a5ae-24962bb4231e-config\") pod \"19bc4d6e-0dc6-461a-a5ae-24962bb4231e\" (UID: \"19bc4d6e-0dc6-461a-a5ae-24962bb4231e\") " Oct 30 10:57:13.524483 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:13.524456 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19bc4d6e-0dc6-461a-a5ae-24962bb4231e-config" pod "19bc4d6e-0dc6-461a-a5ae-24962bb4231e" (UID: "19bc4d6e-0dc6-461a-a5ae-24962bb4231e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:57:13.525953 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:13.525930 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19bc4d6e-0dc6-461a-a5ae-24962bb4231e-tls" pod "19bc4d6e-0dc6-461a-a5ae-24962bb4231e" (UID: "19bc4d6e-0dc6-461a-a5ae-24962bb4231e"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:57:13.526075 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:13.526060 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19bc4d6e-0dc6-461a-a5ae-24962bb4231e-kube-api-access-m5mhr" pod "19bc4d6e-0dc6-461a-a5ae-24962bb4231e" (UID: "19bc4d6e-0dc6-461a-a5ae-24962bb4231e"). InnerVolumeSpecName "kube-api-access-m5mhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:57:13.596350 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:13.596320 4289 generic.go:356] "Generic (PLEG): container finished" podID="19bc4d6e-0dc6-461a-a5ae-24962bb4231e" containerID="771758debfdeff44aa2982abcacdae118af99f714beb6bfa294e8beeaf29649d" exitCode=0 Oct 30 10:57:13.596530 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:13.596377 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr" Oct 30 10:57:13.596530 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:13.596410 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr" event={"ID":"19bc4d6e-0dc6-461a-a5ae-24962bb4231e","Type":"ContainerDied","Data":"771758debfdeff44aa2982abcacdae118af99f714beb6bfa294e8beeaf29649d"} Oct 30 10:57:13.596530 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:13.596492 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr" event={"ID":"19bc4d6e-0dc6-461a-a5ae-24962bb4231e","Type":"ContainerDied","Data":"e659c87dc7e6274320d285b1f8f346182ce573ce43d88aec89e1e6a2f1c68f65"} Oct 30 10:57:13.596530 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:13.596527 4289 scope.go:122] "RemoveContainer" containerID="771758debfdeff44aa2982abcacdae118af99f714beb6bfa294e8beeaf29649d" Oct 30 10:57:13.605585 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:13.605565 4289 scope.go:122] "RemoveContainer" containerID="771758debfdeff44aa2982abcacdae118af99f714beb6bfa294e8beeaf29649d" Oct 30 10:57:13.605829 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:57:13.605810 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"771758debfdeff44aa2982abcacdae118af99f714beb6bfa294e8beeaf29649d\": container with ID starting with 771758debfdeff44aa2982abcacdae118af99f714beb6bfa294e8beeaf29649d not found: ID does not exist" containerID="771758debfdeff44aa2982abcacdae118af99f714beb6bfa294e8beeaf29649d" Oct 30 10:57:13.605886 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:13.605841 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"771758debfdeff44aa2982abcacdae118af99f714beb6bfa294e8beeaf29649d"} err="failed to get container status \"771758debfdeff44aa2982abcacdae118af99f714beb6bfa294e8beeaf29649d\": rpc error: code = NotFound desc = could not find container \"771758debfdeff44aa2982abcacdae118af99f714beb6bfa294e8beeaf29649d\": container with ID starting with 771758debfdeff44aa2982abcacdae118af99f714beb6bfa294e8beeaf29649d not found: ID does not exist" Oct 30 10:57:13.624681 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:13.624653 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/19bc4d6e-0dc6-461a-a5ae-24962bb4231e-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:13.624681 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:13.624675 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5mhr\" (UniqueName: \"kubernetes.io/projected/19bc4d6e-0dc6-461a-a5ae-24962bb4231e-kube-api-access-m5mhr\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:13.624824 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:13.624688 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19bc4d6e-0dc6-461a-a5ae-24962bb4231e-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:13.645414 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:13.645387 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr"] Oct 30 10:57:13.650977 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:13.650956 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-jgmlr"] Oct 30 10:57:14.520832 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:14.520796 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:14.521218 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:14.520964 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="6ec28358-f41b-478f-be0b-b85b3932c116" containerName="otc-container" containerID="cri-o://7dad3636084297a24d0f46fd45718058039dc3646fdb92e46c36a37e36dda4e0" gracePeriod=30 Oct 30 10:57:14.697389 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:14.697354 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:14.782787 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:14.782729 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:14.836401 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:14.836365 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ec28358-f41b-478f-be0b-b85b3932c116-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6ec28358-f41b-478f-be0b-b85b3932c116-data-science-collector-ta-client-cert\") pod \"6ec28358-f41b-478f-be0b-b85b3932c116\" (UID: \"6ec28358-f41b-478f-be0b-b85b3932c116\") " Oct 30 10:57:14.836583 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:14.836451 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ec28358-f41b-478f-be0b-b85b3932c116-kube-api-access-79v42\" (UniqueName: \"kubernetes.io/projected/6ec28358-f41b-478f-be0b-b85b3932c116-kube-api-access-79v42\") pod \"6ec28358-f41b-478f-be0b-b85b3932c116\" (UID: \"6ec28358-f41b-478f-be0b-b85b3932c116\") " Oct 30 10:57:14.836658 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:14.836608 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ec28358-f41b-478f-be0b-b85b3932c116-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec28358-f41b-478f-be0b-b85b3932c116-tls-certs\") pod \"6ec28358-f41b-478f-be0b-b85b3932c116\" (UID: \"6ec28358-f41b-478f-be0b-b85b3932c116\") " Oct 30 10:57:14.836721 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:14.836680 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ec28358-f41b-478f-be0b-b85b3932c116-otc-internal\" (UniqueName: \"kubernetes.io/configmap/6ec28358-f41b-478f-be0b-b85b3932c116-otc-internal\") pod \"6ec28358-f41b-478f-be0b-b85b3932c116\" (UID: \"6ec28358-f41b-478f-be0b-b85b3932c116\") " Oct 30 10:57:14.836801 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:14.836782 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ec28358-f41b-478f-be0b-b85b3932c116-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec28358-f41b-478f-be0b-b85b3932c116-monitoring-tls-certs\") pod \"6ec28358-f41b-478f-be0b-b85b3932c116\" (UID: \"6ec28358-f41b-478f-be0b-b85b3932c116\") " Oct 30 10:57:14.837111 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:14.837060 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec28358-f41b-478f-be0b-b85b3932c116-otc-internal" pod "6ec28358-f41b-478f-be0b-b85b3932c116" (UID: "6ec28358-f41b-478f-be0b-b85b3932c116"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:57:14.838598 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:14.838567 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ec28358-f41b-478f-be0b-b85b3932c116-data-science-collector-ta-client-cert" pod "6ec28358-f41b-478f-be0b-b85b3932c116" (UID: "6ec28358-f41b-478f-be0b-b85b3932c116"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:57:14.838741 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:14.838724 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ec28358-f41b-478f-be0b-b85b3932c116-monitoring-tls-certs" pod "6ec28358-f41b-478f-be0b-b85b3932c116" (UID: "6ec28358-f41b-478f-be0b-b85b3932c116"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:57:14.838960 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:14.838940 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec28358-f41b-478f-be0b-b85b3932c116-kube-api-access-79v42" pod "6ec28358-f41b-478f-be0b-b85b3932c116" (UID: "6ec28358-f41b-478f-be0b-b85b3932c116"). InnerVolumeSpecName "kube-api-access-79v42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:57:14.839018 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:14.838946 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ec28358-f41b-478f-be0b-b85b3932c116-tls-certs" pod "6ec28358-f41b-478f-be0b-b85b3932c116" (UID: "6ec28358-f41b-478f-be0b-b85b3932c116"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:57:14.870557 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:14.869415 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:14.938037 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:14.938008 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79v42\" (UniqueName: \"kubernetes.io/projected/6ec28358-f41b-478f-be0b-b85b3932c116-kube-api-access-79v42\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:14.938037 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:14.938036 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec28358-f41b-478f-be0b-b85b3932c116-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:14.938193 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:14.938047 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6ec28358-f41b-478f-be0b-b85b3932c116-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:14.938193 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:14.938055 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec28358-f41b-478f-be0b-b85b3932c116-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:14.938193 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:14.938066 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6ec28358-f41b-478f-be0b-b85b3932c116-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:15.590715 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.590673 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:15.609164 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.609131 4289 generic.go:356] "Generic (PLEG): container finished" podID="6ec28358-f41b-478f-be0b-b85b3932c116" containerID="7dad3636084297a24d0f46fd45718058039dc3646fdb92e46c36a37e36dda4e0" exitCode=0 Oct 30 10:57:15.609315 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.609183 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6ec28358-f41b-478f-be0b-b85b3932c116","Type":"ContainerDied","Data":"7dad3636084297a24d0f46fd45718058039dc3646fdb92e46c36a37e36dda4e0"} Oct 30 10:57:15.609315 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.609212 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:15.609315 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.609247 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6ec28358-f41b-478f-be0b-b85b3932c116","Type":"ContainerDied","Data":"ca5f479385c6dad3c0b7840e26a5d1f307b62c8fa034fdbc5bd75de565a2d172"} Oct 30 10:57:15.609315 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.609272 4289 scope.go:122] "RemoveContainer" containerID="7dad3636084297a24d0f46fd45718058039dc3646fdb92e46c36a37e36dda4e0" Oct 30 10:57:15.620960 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.620935 4289 scope.go:122] "RemoveContainer" containerID="7dad3636084297a24d0f46fd45718058039dc3646fdb92e46c36a37e36dda4e0" Oct 30 10:57:15.621257 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:57:15.621229 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dad3636084297a24d0f46fd45718058039dc3646fdb92e46c36a37e36dda4e0\": container with ID starting with 7dad3636084297a24d0f46fd45718058039dc3646fdb92e46c36a37e36dda4e0 not found: ID does not exist" containerID="7dad3636084297a24d0f46fd45718058039dc3646fdb92e46c36a37e36dda4e0" Oct 30 10:57:15.621312 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.621269 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dad3636084297a24d0f46fd45718058039dc3646fdb92e46c36a37e36dda4e0"} err="failed to get container status \"7dad3636084297a24d0f46fd45718058039dc3646fdb92e46c36a37e36dda4e0\": rpc error: code = NotFound desc = could not find container \"7dad3636084297a24d0f46fd45718058039dc3646fdb92e46c36a37e36dda4e0\": container with ID starting with 7dad3636084297a24d0f46fd45718058039dc3646fdb92e46c36a37e36dda4e0 not found: ID does not exist" Oct 30 10:57:15.625663 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.625640 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19bc4d6e-0dc6-461a-a5ae-24962bb4231e" path="/var/lib/kubelet/pods/19bc4d6e-0dc6-461a-a5ae-24962bb4231e/volumes" Oct 30 10:57:15.659043 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.659018 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:15.670937 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.670901 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:15.699409 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.699384 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:15.699794 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.699780 4289 cpu_manager.go:414] "containerMap: removing container" podUID="6ec28358-f41b-478f-be0b-b85b3932c116" containerName="otc-container" Oct 30 10:57:15.699834 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.699795 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec28358-f41b-478f-be0b-b85b3932c116" containerName="otc-container" Oct 30 10:57:15.699834 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.699808 4289 cpu_manager.go:414] "containerMap: removing container" podUID="19bc4d6e-0dc6-461a-a5ae-24962bb4231e" containerName="korrel8r" Oct 30 10:57:15.699834 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.699813 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19bc4d6e-0dc6-461a-a5ae-24962bb4231e" containerName="korrel8r" Oct 30 10:57:15.699924 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.699876 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec28358-f41b-478f-be0b-b85b3932c116" containerName="otc-container" Oct 30 10:57:15.699924 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.699884 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="19bc4d6e-0dc6-461a-a5ae-24962bb4231e" containerName="korrel8r" Oct 30 10:57:15.704258 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.704243 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:15.720171 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.720146 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 10:57:15.720306 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.720182 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-e4cadd38\"" Oct 30 10:57:15.720306 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.720210 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8vrjb\"" Oct 30 10:57:15.720306 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.720241 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 10:57:15.720467 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.720324 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 10:57:15.721759 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.721740 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:15.845647 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.845571 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:15.845647 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.845604 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:15.845821 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.845736 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:15.845821 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.845765 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkc2b\" (UniqueName: \"kubernetes.io/projected/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-kube-api-access-xkc2b\") pod \"data-science-collector-collector-1\" (UID: \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:15.845907 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.845829 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:15.946238 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.946201 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:15.946238 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.946236 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:15.946601 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.946294 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:15.946601 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.946320 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkc2b\" (UniqueName: \"kubernetes.io/projected/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-kube-api-access-xkc2b\") pod \"data-science-collector-collector-1\" (UID: \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:15.946601 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.946337 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:15.946901 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.946875 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:15.948715 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.948688 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:15.948944 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.948920 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:15.949008 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.948947 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:15.968741 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:15.968719 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkc2b\" (UniqueName: \"kubernetes.io/projected/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-kube-api-access-xkc2b\") pod \"data-science-collector-collector-1\" (UID: \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:16.016228 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:16.016203 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:16.310071 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:57:16.310043 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79fe2c29_8e9e_4989_828a_6a9bc178a5ca.slice/crio-41650105153f5a27f90cba4477af885b747dbfd4c47ec7cef7c6ce98d6841f4a WatchSource:0}: Error finding container 41650105153f5a27f90cba4477af885b747dbfd4c47ec7cef7c6ce98d6841f4a: Status 404 returned error can't find the container with id 41650105153f5a27f90cba4477af885b747dbfd4c47ec7cef7c6ce98d6841f4a Oct 30 10:57:16.310575 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:16.310551 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:16.312422 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:16.312384 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:57:16.312536 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:16.312458 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:57:16.615996 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:16.615918 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"79fe2c29-8e9e-4989-828a-6a9bc178a5ca","Type":"ContainerStarted","Data":"d96385bf9ff98182acf8a5e32136dda7629a333487336b23e8d742019cc47b6a"} Oct 30 10:57:16.615996 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:16.615972 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"79fe2c29-8e9e-4989-828a-6a9bc178a5ca","Type":"ContainerStarted","Data":"41650105153f5a27f90cba4477af885b747dbfd4c47ec7cef7c6ce98d6841f4a"} Oct 30 10:57:16.662553 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:16.662493 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.662474148 podStartE2EDuration="1.662474148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:57:16.661010641 +0000 UTC m=+727.162390149" watchObservedRunningTime="2026-10-30 10:57:16.662474148 +0000 UTC m=+727.163853633" Oct 30 10:57:17.625067 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:17.625026 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ec28358-f41b-478f-be0b-b85b3932c116" path="/var/lib/kubelet/pods/6ec28358-f41b-478f-be0b-b85b3932c116/volumes" Oct 30 10:57:20.896788 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:20.896753 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:21.096374 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:21.096336 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:22.485717 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:22.485685 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:22.486129 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:22.485891 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="79fe2c29-8e9e-4989-828a-6a9bc178a5ca" containerName="otc-container" containerID="cri-o://d96385bf9ff98182acf8a5e32136dda7629a333487336b23e8d742019cc47b6a" gracePeriod=30 Oct 30 10:57:22.656917 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:22.656883 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:22.839396 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:22.839312 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:23.651170 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:23.651139 4289 generic.go:356] "Generic (PLEG): container finished" podID="79fe2c29-8e9e-4989-828a-6a9bc178a5ca" containerID="d96385bf9ff98182acf8a5e32136dda7629a333487336b23e8d742019cc47b6a" exitCode=0 Oct 30 10:57:23.651556 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:23.651190 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"79fe2c29-8e9e-4989-828a-6a9bc178a5ca","Type":"ContainerDied","Data":"d96385bf9ff98182acf8a5e32136dda7629a333487336b23e8d742019cc47b6a"} Oct 30 10:57:23.732054 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:23.732030 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:23.819066 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:23.818991 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-monitoring-tls-certs\") pod \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\" (UID: \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\") " Oct 30 10:57:23.819066 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:23.819031 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-data-science-collector-ta-client-cert\") pod \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\" (UID: \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\") " Oct 30 10:57:23.819246 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:23.819067 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-kube-api-access-xkc2b\" (UniqueName: \"kubernetes.io/projected/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-kube-api-access-xkc2b\") pod \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\" (UID: \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\") " Oct 30 10:57:23.819246 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:23.819157 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-tls-certs\") pod \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\" (UID: \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\") " Oct 30 10:57:23.819246 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:23.819212 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-otc-internal\" (UniqueName: \"kubernetes.io/configmap/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-otc-internal\") pod \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\" (UID: \"79fe2c29-8e9e-4989-828a-6a9bc178a5ca\") " Oct 30 10:57:23.819680 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:23.819647 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-otc-internal" pod "79fe2c29-8e9e-4989-828a-6a9bc178a5ca" (UID: "79fe2c29-8e9e-4989-828a-6a9bc178a5ca"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:57:23.821361 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:23.821337 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-data-science-collector-ta-client-cert" pod "79fe2c29-8e9e-4989-828a-6a9bc178a5ca" (UID: "79fe2c29-8e9e-4989-828a-6a9bc178a5ca"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:57:23.821361 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:23.821352 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-monitoring-tls-certs" pod "79fe2c29-8e9e-4989-828a-6a9bc178a5ca" (UID: "79fe2c29-8e9e-4989-828a-6a9bc178a5ca"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:57:23.821571 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:23.821379 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-kube-api-access-xkc2b" pod "79fe2c29-8e9e-4989-828a-6a9bc178a5ca" (UID: "79fe2c29-8e9e-4989-828a-6a9bc178a5ca"). InnerVolumeSpecName "kube-api-access-xkc2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:57:23.821571 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:23.821377 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-tls-certs" pod "79fe2c29-8e9e-4989-828a-6a9bc178a5ca" (UID: "79fe2c29-8e9e-4989-828a-6a9bc178a5ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:57:23.920105 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:23.920074 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:23.920105 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:23.920102 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:23.920275 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:23.920113 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:23.920275 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:23.920121 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkc2b\" (UniqueName: \"kubernetes.io/projected/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-kube-api-access-xkc2b\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:23.920275 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:23.920131 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79fe2c29-8e9e-4989-828a-6a9bc178a5ca-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:24.657934 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.657904 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:24.658377 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.657927 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"79fe2c29-8e9e-4989-828a-6a9bc178a5ca","Type":"ContainerDied","Data":"41650105153f5a27f90cba4477af885b747dbfd4c47ec7cef7c6ce98d6841f4a"} Oct 30 10:57:24.658377 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.658008 4289 scope.go:122] "RemoveContainer" containerID="d96385bf9ff98182acf8a5e32136dda7629a333487336b23e8d742019cc47b6a" Oct 30 10:57:24.709333 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.709309 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:24.721306 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.721282 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:24.754267 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.750969 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:24.754267 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.751881 4289 cpu_manager.go:414] "containerMap: removing container" podUID="79fe2c29-8e9e-4989-828a-6a9bc178a5ca" containerName="otc-container" Oct 30 10:57:24.754267 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.751899 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79fe2c29-8e9e-4989-828a-6a9bc178a5ca" containerName="otc-container" Oct 30 10:57:24.754267 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.752144 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="79fe2c29-8e9e-4989-828a-6a9bc178a5ca" containerName="otc-container" Oct 30 10:57:24.757796 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.757775 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:24.772630 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.772575 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:24.772766 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.772739 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 10:57:24.772978 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.772958 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 10:57:24.773073 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.772982 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8vrjb\"" Oct 30 10:57:24.773146 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.773074 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-e0aaab7c\"" Oct 30 10:57:24.773194 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.773150 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 10:57:24.829067 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.829038 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:24.829217 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.829130 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bss6b\" (UniqueName: \"kubernetes.io/projected/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-kube-api-access-bss6b\") pod \"data-science-collector-collector-1\" (UID: \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:24.829217 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.829178 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:24.829294 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.829242 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:24.829294 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.829262 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:24.929728 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.929643 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bss6b\" (UniqueName: \"kubernetes.io/projected/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-kube-api-access-bss6b\") pod \"data-science-collector-collector-1\" (UID: \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:24.929728 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.929693 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:24.929945 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.929745 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:24.929945 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.929768 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:24.929945 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.929792 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:24.930710 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.930686 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:24.932335 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.932308 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:24.932335 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.932333 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:24.932503 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.932311 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:24.952476 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:24.952448 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bss6b\" (UniqueName: \"kubernetes.io/projected/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-kube-api-access-bss6b\") pod \"data-science-collector-collector-1\" (UID: \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:25.069840 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:25.069797 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:25.363790 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:25.363757 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:25.364303 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:57:25.364276 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a6f71a_fab4_4f10_82a5_8f3d4d4fed30.slice/crio-b3fd4a0fc88362f5a8dd6d0ecb8d95e2b611961647917676c823c4e5574c9c79 WatchSource:0}: Error finding container b3fd4a0fc88362f5a8dd6d0ecb8d95e2b611961647917676c823c4e5574c9c79: Status 404 returned error can't find the container with id b3fd4a0fc88362f5a8dd6d0ecb8d95e2b611961647917676c823c4e5574c9c79 Oct 30 10:57:25.366408 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:25.366374 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:57:25.366517 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:25.366448 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:57:25.625987 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:25.625895 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79fe2c29-8e9e-4989-828a-6a9bc178a5ca" path="/var/lib/kubelet/pods/79fe2c29-8e9e-4989-828a-6a9bc178a5ca/volumes" Oct 30 10:57:25.665421 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:25.665386 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30","Type":"ContainerStarted","Data":"6cea8e1afe99dd52e0d8633733fc11ab3747b4b58fbc99dd829247459c6515f1"} Oct 30 10:57:25.665800 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:25.665449 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30","Type":"ContainerStarted","Data":"b3fd4a0fc88362f5a8dd6d0ecb8d95e2b611961647917676c823c4e5574c9c79"} Oct 30 10:57:25.707468 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:25.707400 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.707387925 podStartE2EDuration="1.707387925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:57:25.706020681 +0000 UTC m=+736.207400166" watchObservedRunningTime="2026-10-30 10:57:25.707387925 +0000 UTC m=+736.208767409" Oct 30 10:57:26.257813 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:57:26.257778 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:57:28.633579 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:28.633538 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:28.834631 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:28.834580 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:28.994977 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:28.994942 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:30.307006 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.306974 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:30.307488 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.307152 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="59a6f71a-fab4-4f10-82a5-8f3d4d4fed30" containerName="otc-container" containerID="cri-o://6cea8e1afe99dd52e0d8633733fc11ab3747b4b58fbc99dd829247459c6515f1" gracePeriod=30 Oct 30 10:57:30.489399 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.489357 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:30.555985 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.555957 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:30.652179 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.652081 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:30.681190 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.681151 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-monitoring-tls-certs\") pod \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\" (UID: \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\") " Oct 30 10:57:30.681347 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.681235 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-kube-api-access-bss6b\" (UniqueName: \"kubernetes.io/projected/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-kube-api-access-bss6b\") pod \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\" (UID: \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\") " Oct 30 10:57:30.681347 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.681257 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-tls-certs\" (UniqueName: \"kubernetes.io/secret/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-tls-certs\") pod \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\" (UID: \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\") " Oct 30 10:57:30.681347 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.681285 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-data-science-collector-ta-client-cert\") pod \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\" (UID: \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\") " Oct 30 10:57:30.681347 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.681327 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-otc-internal\" (UniqueName: \"kubernetes.io/configmap/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-otc-internal\") pod \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\" (UID: \"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30\") " Oct 30 10:57:30.681744 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.681715 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-otc-internal" pod "59a6f71a-fab4-4f10-82a5-8f3d4d4fed30" (UID: "59a6f71a-fab4-4f10-82a5-8f3d4d4fed30"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:57:30.683424 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.683398 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-tls-certs" pod "59a6f71a-fab4-4f10-82a5-8f3d4d4fed30" (UID: "59a6f71a-fab4-4f10-82a5-8f3d4d4fed30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:57:30.683537 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.683417 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-data-science-collector-ta-client-cert" pod "59a6f71a-fab4-4f10-82a5-8f3d4d4fed30" (UID: "59a6f71a-fab4-4f10-82a5-8f3d4d4fed30"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:57:30.683736 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.683713 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-monitoring-tls-certs" pod "59a6f71a-fab4-4f10-82a5-8f3d4d4fed30" (UID: "59a6f71a-fab4-4f10-82a5-8f3d4d4fed30"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:57:30.683801 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.683729 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-kube-api-access-bss6b" pod "59a6f71a-fab4-4f10-82a5-8f3d4d4fed30" (UID: "59a6f71a-fab4-4f10-82a5-8f3d4d4fed30"). InnerVolumeSpecName "kube-api-access-bss6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:57:30.690609 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.690583 4289 generic.go:356] "Generic (PLEG): container finished" podID="59a6f71a-fab4-4f10-82a5-8f3d4d4fed30" containerID="6cea8e1afe99dd52e0d8633733fc11ab3747b4b58fbc99dd829247459c6515f1" exitCode=0 Oct 30 10:57:30.690660 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.690635 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:30.690702 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.690664 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30","Type":"ContainerDied","Data":"6cea8e1afe99dd52e0d8633733fc11ab3747b4b58fbc99dd829247459c6515f1"} Oct 30 10:57:30.690737 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.690727 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"59a6f71a-fab4-4f10-82a5-8f3d4d4fed30","Type":"ContainerDied","Data":"b3fd4a0fc88362f5a8dd6d0ecb8d95e2b611961647917676c823c4e5574c9c79"} Oct 30 10:57:30.690777 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.690757 4289 scope.go:122] "RemoveContainer" containerID="6cea8e1afe99dd52e0d8633733fc11ab3747b4b58fbc99dd829247459c6515f1" Oct 30 10:57:30.703693 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.703675 4289 scope.go:122] "RemoveContainer" containerID="6cea8e1afe99dd52e0d8633733fc11ab3747b4b58fbc99dd829247459c6515f1" Oct 30 10:57:30.703971 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:57:30.703943 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cea8e1afe99dd52e0d8633733fc11ab3747b4b58fbc99dd829247459c6515f1\": container with ID starting with 6cea8e1afe99dd52e0d8633733fc11ab3747b4b58fbc99dd829247459c6515f1 not found: ID does not exist" containerID="6cea8e1afe99dd52e0d8633733fc11ab3747b4b58fbc99dd829247459c6515f1" Oct 30 10:57:30.704038 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.703982 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cea8e1afe99dd52e0d8633733fc11ab3747b4b58fbc99dd829247459c6515f1"} err="failed to get container status \"6cea8e1afe99dd52e0d8633733fc11ab3747b4b58fbc99dd829247459c6515f1\": rpc error: code = NotFound desc = could not find container \"6cea8e1afe99dd52e0d8633733fc11ab3747b4b58fbc99dd829247459c6515f1\": container with ID starting with 6cea8e1afe99dd52e0d8633733fc11ab3747b4b58fbc99dd829247459c6515f1 not found: ID does not exist" Oct 30 10:57:30.739384 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.739358 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:30.745161 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.745137 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:30.782305 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.782279 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bss6b\" (UniqueName: \"kubernetes.io/projected/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-kube-api-access-bss6b\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:30.782305 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.782291 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:30.782478 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.782301 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:30.782478 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.782398 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:30.782478 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.782407 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:30.782478 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.782417 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:30.782918 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.782902 4289 cpu_manager.go:414] "containerMap: removing container" podUID="59a6f71a-fab4-4f10-82a5-8f3d4d4fed30" containerName="otc-container" Oct 30 10:57:30.782953 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.782921 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59a6f71a-fab4-4f10-82a5-8f3d4d4fed30" containerName="otc-container" Oct 30 10:57:30.783042 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.783031 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="59a6f71a-fab4-4f10-82a5-8f3d4d4fed30" containerName="otc-container" Oct 30 10:57:30.787795 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.787780 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:30.804094 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.804072 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-e4cadd38\"" Oct 30 10:57:30.804094 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.804101 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8vrjb\"" Oct 30 10:57:30.804094 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.804182 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 10:57:30.804094 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.804243 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 10:57:30.804094 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.804282 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 10:57:30.805810 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.805789 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:30.883733 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.883702 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8f2c2bba-7dbd-42d0-887e-5092cac04648-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"8f2c2bba-7dbd-42d0-887e-5092cac04648\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:30.883863 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.883753 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f2c2bba-7dbd-42d0-887e-5092cac04648-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8f2c2bba-7dbd-42d0-887e-5092cac04648\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:30.883863 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.883850 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8f2c2bba-7dbd-42d0-887e-5092cac04648-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"8f2c2bba-7dbd-42d0-887e-5092cac04648\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:30.883950 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.883894 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw2jz\" (UniqueName: \"kubernetes.io/projected/8f2c2bba-7dbd-42d0-887e-5092cac04648-kube-api-access-sw2jz\") pod \"data-science-collector-collector-1\" (UID: \"8f2c2bba-7dbd-42d0-887e-5092cac04648\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:30.883950 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.883911 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f2c2bba-7dbd-42d0-887e-5092cac04648-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8f2c2bba-7dbd-42d0-887e-5092cac04648\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:30.984727 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.984695 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8f2c2bba-7dbd-42d0-887e-5092cac04648-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"8f2c2bba-7dbd-42d0-887e-5092cac04648\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:30.984875 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.984746 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f2c2bba-7dbd-42d0-887e-5092cac04648-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8f2c2bba-7dbd-42d0-887e-5092cac04648\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:30.984875 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.984772 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8f2c2bba-7dbd-42d0-887e-5092cac04648-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"8f2c2bba-7dbd-42d0-887e-5092cac04648\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:30.984875 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.984812 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw2jz\" (UniqueName: \"kubernetes.io/projected/8f2c2bba-7dbd-42d0-887e-5092cac04648-kube-api-access-sw2jz\") pod \"data-science-collector-collector-1\" (UID: \"8f2c2bba-7dbd-42d0-887e-5092cac04648\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:30.984875 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.984830 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f2c2bba-7dbd-42d0-887e-5092cac04648-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8f2c2bba-7dbd-42d0-887e-5092cac04648\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:30.985359 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.985331 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8f2c2bba-7dbd-42d0-887e-5092cac04648-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"8f2c2bba-7dbd-42d0-887e-5092cac04648\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:30.987249 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.987219 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f2c2bba-7dbd-42d0-887e-5092cac04648-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8f2c2bba-7dbd-42d0-887e-5092cac04648\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:30.987376 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.987287 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f2c2bba-7dbd-42d0-887e-5092cac04648-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8f2c2bba-7dbd-42d0-887e-5092cac04648\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:30.987470 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:30.987448 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8f2c2bba-7dbd-42d0-887e-5092cac04648-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"8f2c2bba-7dbd-42d0-887e-5092cac04648\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:31.006954 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:31.006928 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw2jz\" (UniqueName: \"kubernetes.io/projected/8f2c2bba-7dbd-42d0-887e-5092cac04648-kube-api-access-sw2jz\") pod \"data-science-collector-collector-1\" (UID: \"8f2c2bba-7dbd-42d0-887e-5092cac04648\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:31.100179 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:31.100147 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:31.392974 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:31.392941 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:31.393943 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:57:31.393913 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f2c2bba_7dbd_42d0_887e_5092cac04648.slice/crio-8b90bc6777d5fa998cc3cca2f9b79693d3df00eb58725a737c5aaa4a099b192c WatchSource:0}: Error finding container 8b90bc6777d5fa998cc3cca2f9b79693d3df00eb58725a737c5aaa4a099b192c: Status 404 returned error can't find the container with id 8b90bc6777d5fa998cc3cca2f9b79693d3df00eb58725a737c5aaa4a099b192c Oct 30 10:57:31.396031 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:31.395997 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:57:31.396125 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:31.396066 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:57:31.625619 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:31.625579 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59a6f71a-fab4-4f10-82a5-8f3d4d4fed30" path="/var/lib/kubelet/pods/59a6f71a-fab4-4f10-82a5-8f3d4d4fed30/volumes" Oct 30 10:57:31.696644 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:31.696556 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8f2c2bba-7dbd-42d0-887e-5092cac04648","Type":"ContainerStarted","Data":"9157c478c43caf75da3272b79a89a60b860583ef15cdca5d3394af859c978888"} Oct 30 10:57:31.696644 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:31.696611 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8f2c2bba-7dbd-42d0-887e-5092cac04648","Type":"ContainerStarted","Data":"8b90bc6777d5fa998cc3cca2f9b79693d3df00eb58725a737c5aaa4a099b192c"} Oct 30 10:57:31.740675 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:31.740624 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.7406115899999999 podStartE2EDuration="1.74061159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:57:31.73941243 +0000 UTC m=+742.240791915" watchObservedRunningTime="2026-10-30 10:57:31.74061159 +0000 UTC m=+742.241991075" Oct 30 10:57:36.747649 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:36.747610 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:36.918761 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:36.918730 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:37.089379 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:37.089289 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:37.106463 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:37.106416 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml"] Oct 30 10:57:37.110176 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:37.110155 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml" Oct 30 10:57:37.128920 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:37.128896 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml"] Oct 30 10:57:37.238698 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:37.238669 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdm9l\" (UniqueName: \"kubernetes.io/projected/5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f-kube-api-access-fdm9l\") pod \"korrel8r-66d6f7f696-4sdml\" (UID: \"5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml" Oct 30 10:57:37.238878 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:37.238705 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f-tls\") pod \"korrel8r-66d6f7f696-4sdml\" (UID: \"5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml" Oct 30 10:57:37.238878 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:37.238738 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f-config\") pod \"korrel8r-66d6f7f696-4sdml\" (UID: \"5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml" Oct 30 10:57:37.340316 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:37.340239 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdm9l\" (UniqueName: \"kubernetes.io/projected/5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f-kube-api-access-fdm9l\") pod \"korrel8r-66d6f7f696-4sdml\" (UID: \"5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml" Oct 30 10:57:37.340316 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:37.340275 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f-tls\") pod \"korrel8r-66d6f7f696-4sdml\" (UID: \"5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml" Oct 30 10:57:37.340531 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:37.340328 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f-config\") pod \"korrel8r-66d6f7f696-4sdml\" (UID: \"5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml" Oct 30 10:57:37.341145 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:37.341120 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f-config\") pod \"korrel8r-66d6f7f696-4sdml\" (UID: \"5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml" Oct 30 10:57:37.342763 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:37.342744 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f-tls\") pod \"korrel8r-66d6f7f696-4sdml\" (UID: \"5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml" Oct 30 10:57:37.362804 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:37.362778 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdm9l\" (UniqueName: \"kubernetes.io/projected/5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f-kube-api-access-fdm9l\") pod \"korrel8r-66d6f7f696-4sdml\" (UID: \"5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml" Oct 30 10:57:37.422793 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:37.422762 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml" Oct 30 10:57:37.719397 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:37.719370 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml"] Oct 30 10:57:37.721136 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:57:37.721110 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6b4edf_57b0_4ac9_848d_d7abd5f9ed0f.slice/crio-15c0947e52e44eb9033c42442afc37004e4b55ac500676ed00e5814f34f91af1 WatchSource:0}: Error finding container 15c0947e52e44eb9033c42442afc37004e4b55ac500676ed00e5814f34f91af1: Status 404 returned error can't find the container with id 15c0947e52e44eb9033c42442afc37004e4b55ac500676ed00e5814f34f91af1 Oct 30 10:57:37.728819 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:37.728791 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml" event={"ID":"5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f","Type":"ContainerStarted","Data":"15c0947e52e44eb9033c42442afc37004e4b55ac500676ed00e5814f34f91af1"} Oct 30 10:57:38.525502 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.525472 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:38.525890 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.525660 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="8f2c2bba-7dbd-42d0-887e-5092cac04648" containerName="otc-container" containerID="cri-o://9157c478c43caf75da3272b79a89a60b860583ef15cdca5d3394af859c978888" gracePeriod=30 Oct 30 10:57:38.697853 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.697693 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:38.736082 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.736048 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml" event={"ID":"5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f","Type":"ContainerStarted","Data":"03a7d046487c302ef2ea8c5607aa6f51bd9d92dc67eb557114d2a06520fa536b"} Oct 30 10:57:38.736294 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.736239 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml" Oct 30 10:57:38.737800 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.737774 4289 generic.go:356] "Generic (PLEG): container finished" podID="8f2c2bba-7dbd-42d0-887e-5092cac04648" containerID="9157c478c43caf75da3272b79a89a60b860583ef15cdca5d3394af859c978888" exitCode=0 Oct 30 10:57:38.737918 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.737830 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8f2c2bba-7dbd-42d0-887e-5092cac04648","Type":"ContainerDied","Data":"9157c478c43caf75da3272b79a89a60b860583ef15cdca5d3394af859c978888"} Oct 30 10:57:38.780700 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.780630 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:38.782278 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.782232 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml" podStartSLOduration=1.782219332 podStartE2EDuration="1.782219332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:57:38.780620815 +0000 UTC m=+749.282000300" watchObservedRunningTime="2026-10-30 10:57:38.782219332 +0000 UTC m=+749.283598844" Oct 30 10:57:38.854407 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.854379 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f2c2bba-7dbd-42d0-887e-5092cac04648-otc-internal\" (UniqueName: \"kubernetes.io/configmap/8f2c2bba-7dbd-42d0-887e-5092cac04648-otc-internal\") pod \"8f2c2bba-7dbd-42d0-887e-5092cac04648\" (UID: \"8f2c2bba-7dbd-42d0-887e-5092cac04648\") " Oct 30 10:57:38.854561 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.854456 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f2c2bba-7dbd-42d0-887e-5092cac04648-kube-api-access-sw2jz\" (UniqueName: \"kubernetes.io/projected/8f2c2bba-7dbd-42d0-887e-5092cac04648-kube-api-access-sw2jz\") pod \"8f2c2bba-7dbd-42d0-887e-5092cac04648\" (UID: \"8f2c2bba-7dbd-42d0-887e-5092cac04648\") " Oct 30 10:57:38.854561 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.854486 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f2c2bba-7dbd-42d0-887e-5092cac04648-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8f2c2bba-7dbd-42d0-887e-5092cac04648-data-science-collector-ta-client-cert\") pod \"8f2c2bba-7dbd-42d0-887e-5092cac04648\" (UID: \"8f2c2bba-7dbd-42d0-887e-5092cac04648\") " Oct 30 10:57:38.854561 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.854513 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f2c2bba-7dbd-42d0-887e-5092cac04648-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f2c2bba-7dbd-42d0-887e-5092cac04648-tls-certs\") pod \"8f2c2bba-7dbd-42d0-887e-5092cac04648\" (UID: \"8f2c2bba-7dbd-42d0-887e-5092cac04648\") " Oct 30 10:57:38.854561 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.854556 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f2c2bba-7dbd-42d0-887e-5092cac04648-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f2c2bba-7dbd-42d0-887e-5092cac04648-monitoring-tls-certs\") pod \"8f2c2bba-7dbd-42d0-887e-5092cac04648\" (UID: \"8f2c2bba-7dbd-42d0-887e-5092cac04648\") " Oct 30 10:57:38.854801 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.854775 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f2c2bba-7dbd-42d0-887e-5092cac04648-otc-internal" pod "8f2c2bba-7dbd-42d0-887e-5092cac04648" (UID: "8f2c2bba-7dbd-42d0-887e-5092cac04648"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:57:38.854954 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.854929 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8f2c2bba-7dbd-42d0-887e-5092cac04648-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:38.856585 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.856555 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f2c2bba-7dbd-42d0-887e-5092cac04648-tls-certs" pod "8f2c2bba-7dbd-42d0-887e-5092cac04648" (UID: "8f2c2bba-7dbd-42d0-887e-5092cac04648"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:57:38.856717 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.856693 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2c2bba-7dbd-42d0-887e-5092cac04648-kube-api-access-sw2jz" pod "8f2c2bba-7dbd-42d0-887e-5092cac04648" (UID: "8f2c2bba-7dbd-42d0-887e-5092cac04648"). InnerVolumeSpecName "kube-api-access-sw2jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:57:38.857004 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.856979 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f2c2bba-7dbd-42d0-887e-5092cac04648-data-science-collector-ta-client-cert" pod "8f2c2bba-7dbd-42d0-887e-5092cac04648" (UID: "8f2c2bba-7dbd-42d0-887e-5092cac04648"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:57:38.857004 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.856997 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f2c2bba-7dbd-42d0-887e-5092cac04648-monitoring-tls-certs" pod "8f2c2bba-7dbd-42d0-887e-5092cac04648" (UID: "8f2c2bba-7dbd-42d0-887e-5092cac04648"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:57:38.863660 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.863604 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:38.956399 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.956370 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f2c2bba-7dbd-42d0-887e-5092cac04648-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:38.956399 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.956396 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f2c2bba-7dbd-42d0-887e-5092cac04648-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:38.956399 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.956406 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw2jz\" (UniqueName: \"kubernetes.io/projected/8f2c2bba-7dbd-42d0-887e-5092cac04648-kube-api-access-sw2jz\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:38.956613 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:38.956416 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8f2c2bba-7dbd-42d0-887e-5092cac04648-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:39.744111 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:39.744082 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:39.744511 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:39.744124 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8f2c2bba-7dbd-42d0-887e-5092cac04648","Type":"ContainerDied","Data":"8b90bc6777d5fa998cc3cca2f9b79693d3df00eb58725a737c5aaa4a099b192c"} Oct 30 10:57:39.744511 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:39.744174 4289 scope.go:122] "RemoveContainer" containerID="9157c478c43caf75da3272b79a89a60b860583ef15cdca5d3394af859c978888" Oct 30 10:57:39.793893 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:39.793865 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:39.805212 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:39.805189 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:39.835740 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:39.835714 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:39.836319 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:39.836296 4289 cpu_manager.go:414] "containerMap: removing container" podUID="8f2c2bba-7dbd-42d0-887e-5092cac04648" containerName="otc-container" Oct 30 10:57:39.836319 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:39.836318 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f2c2bba-7dbd-42d0-887e-5092cac04648" containerName="otc-container" Oct 30 10:57:39.836519 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:39.836420 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f2c2bba-7dbd-42d0-887e-5092cac04648" containerName="otc-container" Oct 30 10:57:39.841272 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:39.841253 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:39.857703 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:39.857633 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 10:57:39.857703 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:39.857675 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 10:57:39.857703 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:39.857687 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-4873cf1a\"" Oct 30 10:57:39.857894 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:39.857771 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8vrjb\"" Oct 30 10:57:39.857931 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:39.857910 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 10:57:39.885346 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:39.885295 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:39.964106 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:39.964076 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5c814605-a67a-42fa-8e9c-af9c352ff482-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5c814605-a67a-42fa-8e9c-af9c352ff482\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:39.964106 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:39.964106 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c814605-a67a-42fa-8e9c-af9c352ff482-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5c814605-a67a-42fa-8e9c-af9c352ff482\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:39.964299 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:39.964215 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtd5l\" (UniqueName: \"kubernetes.io/projected/5c814605-a67a-42fa-8e9c-af9c352ff482-kube-api-access-jtd5l\") pod \"data-science-collector-collector-1\" (UID: \"5c814605-a67a-42fa-8e9c-af9c352ff482\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:39.964299 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:39.964255 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5c814605-a67a-42fa-8e9c-af9c352ff482-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5c814605-a67a-42fa-8e9c-af9c352ff482\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:39.964299 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:39.964276 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c814605-a67a-42fa-8e9c-af9c352ff482-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5c814605-a67a-42fa-8e9c-af9c352ff482\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:40.065034 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:40.064982 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5c814605-a67a-42fa-8e9c-af9c352ff482-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5c814605-a67a-42fa-8e9c-af9c352ff482\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:40.065034 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:40.065034 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c814605-a67a-42fa-8e9c-af9c352ff482-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5c814605-a67a-42fa-8e9c-af9c352ff482\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:40.065242 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:40.065130 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtd5l\" (UniqueName: \"kubernetes.io/projected/5c814605-a67a-42fa-8e9c-af9c352ff482-kube-api-access-jtd5l\") pod \"data-science-collector-collector-1\" (UID: \"5c814605-a67a-42fa-8e9c-af9c352ff482\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:40.065242 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:40.065163 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5c814605-a67a-42fa-8e9c-af9c352ff482-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5c814605-a67a-42fa-8e9c-af9c352ff482\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:40.065242 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:40.065189 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c814605-a67a-42fa-8e9c-af9c352ff482-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5c814605-a67a-42fa-8e9c-af9c352ff482\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:40.065828 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:40.065805 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5c814605-a67a-42fa-8e9c-af9c352ff482-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5c814605-a67a-42fa-8e9c-af9c352ff482\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:40.067571 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:40.067548 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c814605-a67a-42fa-8e9c-af9c352ff482-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5c814605-a67a-42fa-8e9c-af9c352ff482\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:40.067682 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:40.067598 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c814605-a67a-42fa-8e9c-af9c352ff482-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5c814605-a67a-42fa-8e9c-af9c352ff482\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:40.067728 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:40.067674 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5c814605-a67a-42fa-8e9c-af9c352ff482-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5c814605-a67a-42fa-8e9c-af9c352ff482\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:40.087569 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:40.087543 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtd5l\" (UniqueName: \"kubernetes.io/projected/5c814605-a67a-42fa-8e9c-af9c352ff482-kube-api-access-jtd5l\") pod \"data-science-collector-collector-1\" (UID: \"5c814605-a67a-42fa-8e9c-af9c352ff482\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:40.155618 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:40.155543 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:40.311708 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:40.311656 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:40.455175 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:40.455145 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:40.455423 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:57:40.455402 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c814605_a67a_42fa_8e9c_af9c352ff482.slice/crio-d91450c6db508674ace2d0a07e52543ffc4df1925412511965a5c50f273655c4 WatchSource:0}: Error finding container d91450c6db508674ace2d0a07e52543ffc4df1925412511965a5c50f273655c4: Status 404 returned error can't find the container with id d91450c6db508674ace2d0a07e52543ffc4df1925412511965a5c50f273655c4 Oct 30 10:57:40.458098 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:40.458066 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:57:40.458169 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:40.458120 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:57:40.624260 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:40.624230 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:40.751150 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:40.751118 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5c814605-a67a-42fa-8e9c-af9c352ff482","Type":"ContainerStarted","Data":"5c2fea0fbb16269a6f61d9555178808e2e8c7b3303046c89190d8ca9a3715d81"} Oct 30 10:57:40.751601 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:40.751174 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5c814605-a67a-42fa-8e9c-af9c352ff482","Type":"ContainerStarted","Data":"d91450c6db508674ace2d0a07e52543ffc4df1925412511965a5c50f273655c4"} Oct 30 10:57:40.794445 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:40.794389 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.7943745070000001 podStartE2EDuration="1.794374507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:57:40.793649453 +0000 UTC m=+751.295028974" watchObservedRunningTime="2026-10-30 10:57:40.794374507 +0000 UTC m=+751.295753991" Oct 30 10:57:41.627532 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:41.627493 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f2c2bba-7dbd-42d0-887e-5092cac04648" path="/var/lib/kubelet/pods/8f2c2bba-7dbd-42d0-887e-5092cac04648/volumes" Oct 30 10:57:43.778150 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:43.778116 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:45.518393 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:45.518357 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:45.723186 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:45.723148 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:45.839457 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:45.839356 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml"] Oct 30 10:57:45.839617 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:45.839559 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml" podUID="5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f" containerName="korrel8r" containerID="cri-o://03a7d046487c302ef2ea8c5607aa6f51bd9d92dc67eb557114d2a06520fa536b" gracePeriod=30 Oct 30 10:57:45.841686 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:45.841660 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml" Oct 30 10:57:45.889800 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:45.889770 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:46.097559 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:46.097505 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml" Oct 30 10:57:46.223651 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:46.223621 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f-kube-api-access-fdm9l\" (UniqueName: \"kubernetes.io/projected/5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f-kube-api-access-fdm9l\") pod \"5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f\" (UID: \"5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f\") " Oct 30 10:57:46.223811 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:46.223704 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f-config\" (UniqueName: \"kubernetes.io/configmap/5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f-config\") pod \"5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f\" (UID: \"5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f\") " Oct 30 10:57:46.223872 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:46.223806 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f-tls\" (UniqueName: \"kubernetes.io/secret/5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f-tls\") pod \"5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f\" (UID: \"5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f\") " Oct 30 10:57:46.224087 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:46.224060 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f-config" pod "5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f" (UID: "5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:57:46.225758 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:46.225733 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f-kube-api-access-fdm9l" pod "5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f" (UID: "5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f"). InnerVolumeSpecName "kube-api-access-fdm9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:57:46.225834 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:46.225789 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f-tls" pod "5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f" (UID: "5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:57:46.325108 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:46.325068 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdm9l\" (UniqueName: \"kubernetes.io/projected/5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f-kube-api-access-fdm9l\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:46.325108 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:46.325101 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:46.325108 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:46.325114 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:46.783468 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:46.783406 4289 generic.go:356] "Generic (PLEG): container finished" podID="5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f" containerID="03a7d046487c302ef2ea8c5607aa6f51bd9d92dc67eb557114d2a06520fa536b" exitCode=0 Oct 30 10:57:46.783991 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:46.783470 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml" event={"ID":"5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f","Type":"ContainerDied","Data":"03a7d046487c302ef2ea8c5607aa6f51bd9d92dc67eb557114d2a06520fa536b"} Oct 30 10:57:46.783991 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:46.783523 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml" event={"ID":"5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f","Type":"ContainerDied","Data":"15c0947e52e44eb9033c42442afc37004e4b55ac500676ed00e5814f34f91af1"} Oct 30 10:57:46.783991 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:46.783545 4289 scope.go:122] "RemoveContainer" containerID="03a7d046487c302ef2ea8c5607aa6f51bd9d92dc67eb557114d2a06520fa536b" Oct 30 10:57:46.783991 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:46.783484 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml" Oct 30 10:57:46.792217 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:46.792199 4289 scope.go:122] "RemoveContainer" containerID="03a7d046487c302ef2ea8c5607aa6f51bd9d92dc67eb557114d2a06520fa536b" Oct 30 10:57:46.792505 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:57:46.792480 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03a7d046487c302ef2ea8c5607aa6f51bd9d92dc67eb557114d2a06520fa536b\": container with ID starting with 03a7d046487c302ef2ea8c5607aa6f51bd9d92dc67eb557114d2a06520fa536b not found: ID does not exist" containerID="03a7d046487c302ef2ea8c5607aa6f51bd9d92dc67eb557114d2a06520fa536b" Oct 30 10:57:46.792605 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:46.792510 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03a7d046487c302ef2ea8c5607aa6f51bd9d92dc67eb557114d2a06520fa536b"} err="failed to get container status \"03a7d046487c302ef2ea8c5607aa6f51bd9d92dc67eb557114d2a06520fa536b\": rpc error: code = NotFound desc = could not find container \"03a7d046487c302ef2ea8c5607aa6f51bd9d92dc67eb557114d2a06520fa536b\": container with ID starting with 03a7d046487c302ef2ea8c5607aa6f51bd9d92dc67eb557114d2a06520fa536b not found: ID does not exist" Oct 30 10:57:46.832624 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:46.832597 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml"] Oct 30 10:57:46.844327 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:46.844303 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-4sdml"] Oct 30 10:57:47.190762 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.190681 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:47.190912 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.190851 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="5c814605-a67a-42fa-8e9c-af9c352ff482" containerName="otc-container" containerID="cri-o://5c2fea0fbb16269a6f61d9555178808e2e8c7b3303046c89190d8ca9a3715d81" gracePeriod=30 Oct 30 10:57:47.366946 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.366907 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:47.444001 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.443944 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:47.539597 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.539561 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:47.624706 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.624675 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f" path="/var/lib/kubelet/pods/5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f/volumes" Oct 30 10:57:47.636711 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.636689 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c814605-a67a-42fa-8e9c-af9c352ff482-kube-api-access-jtd5l\" (UniqueName: \"kubernetes.io/projected/5c814605-a67a-42fa-8e9c-af9c352ff482-kube-api-access-jtd5l\") pod \"5c814605-a67a-42fa-8e9c-af9c352ff482\" (UID: \"5c814605-a67a-42fa-8e9c-af9c352ff482\") " Oct 30 10:57:47.636839 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.636731 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c814605-a67a-42fa-8e9c-af9c352ff482-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c814605-a67a-42fa-8e9c-af9c352ff482-tls-certs\") pod \"5c814605-a67a-42fa-8e9c-af9c352ff482\" (UID: \"5c814605-a67a-42fa-8e9c-af9c352ff482\") " Oct 30 10:57:47.636901 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.636867 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c814605-a67a-42fa-8e9c-af9c352ff482-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c814605-a67a-42fa-8e9c-af9c352ff482-monitoring-tls-certs\") pod \"5c814605-a67a-42fa-8e9c-af9c352ff482\" (UID: \"5c814605-a67a-42fa-8e9c-af9c352ff482\") " Oct 30 10:57:47.636965 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.636947 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c814605-a67a-42fa-8e9c-af9c352ff482-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5c814605-a67a-42fa-8e9c-af9c352ff482-otc-internal\") pod \"5c814605-a67a-42fa-8e9c-af9c352ff482\" (UID: \"5c814605-a67a-42fa-8e9c-af9c352ff482\") " Oct 30 10:57:47.637021 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.637003 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c814605-a67a-42fa-8e9c-af9c352ff482-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5c814605-a67a-42fa-8e9c-af9c352ff482-data-science-collector-ta-client-cert\") pod \"5c814605-a67a-42fa-8e9c-af9c352ff482\" (UID: \"5c814605-a67a-42fa-8e9c-af9c352ff482\") " Oct 30 10:57:47.637364 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.637321 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c814605-a67a-42fa-8e9c-af9c352ff482-otc-internal" pod "5c814605-a67a-42fa-8e9c-af9c352ff482" (UID: "5c814605-a67a-42fa-8e9c-af9c352ff482"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:57:47.639152 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.639121 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c814605-a67a-42fa-8e9c-af9c352ff482-tls-certs" pod "5c814605-a67a-42fa-8e9c-af9c352ff482" (UID: "5c814605-a67a-42fa-8e9c-af9c352ff482"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:57:47.639380 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.639357 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c814605-a67a-42fa-8e9c-af9c352ff482-monitoring-tls-certs" pod "5c814605-a67a-42fa-8e9c-af9c352ff482" (UID: "5c814605-a67a-42fa-8e9c-af9c352ff482"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:57:47.639474 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.639415 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c814605-a67a-42fa-8e9c-af9c352ff482-data-science-collector-ta-client-cert" pod "5c814605-a67a-42fa-8e9c-af9c352ff482" (UID: "5c814605-a67a-42fa-8e9c-af9c352ff482"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:57:47.639474 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.639444 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c814605-a67a-42fa-8e9c-af9c352ff482-kube-api-access-jtd5l" pod "5c814605-a67a-42fa-8e9c-af9c352ff482" (UID: "5c814605-a67a-42fa-8e9c-af9c352ff482"). InnerVolumeSpecName "kube-api-access-jtd5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:57:47.738007 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.737980 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5c814605-a67a-42fa-8e9c-af9c352ff482-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:47.738148 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.738008 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5c814605-a67a-42fa-8e9c-af9c352ff482-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:47.738148 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.738043 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtd5l\" (UniqueName: \"kubernetes.io/projected/5c814605-a67a-42fa-8e9c-af9c352ff482-kube-api-access-jtd5l\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:47.738148 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.738053 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c814605-a67a-42fa-8e9c-af9c352ff482-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:47.738148 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.738062 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c814605-a67a-42fa-8e9c-af9c352ff482-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:47.794632 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.794595 4289 generic.go:356] "Generic (PLEG): container finished" podID="5c814605-a67a-42fa-8e9c-af9c352ff482" containerID="5c2fea0fbb16269a6f61d9555178808e2e8c7b3303046c89190d8ca9a3715d81" exitCode=0 Oct 30 10:57:47.795049 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.794643 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5c814605-a67a-42fa-8e9c-af9c352ff482","Type":"ContainerDied","Data":"5c2fea0fbb16269a6f61d9555178808e2e8c7b3303046c89190d8ca9a3715d81"} Oct 30 10:57:47.795049 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.794677 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:47.795049 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.794717 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5c814605-a67a-42fa-8e9c-af9c352ff482","Type":"ContainerDied","Data":"d91450c6db508674ace2d0a07e52543ffc4df1925412511965a5c50f273655c4"} Oct 30 10:57:47.795049 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.794752 4289 scope.go:122] "RemoveContainer" containerID="5c2fea0fbb16269a6f61d9555178808e2e8c7b3303046c89190d8ca9a3715d81" Oct 30 10:57:47.804240 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.804221 4289 scope.go:122] "RemoveContainer" containerID="5c2fea0fbb16269a6f61d9555178808e2e8c7b3303046c89190d8ca9a3715d81" Oct 30 10:57:47.804558 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:57:47.804537 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c2fea0fbb16269a6f61d9555178808e2e8c7b3303046c89190d8ca9a3715d81\": container with ID starting with 5c2fea0fbb16269a6f61d9555178808e2e8c7b3303046c89190d8ca9a3715d81 not found: ID does not exist" containerID="5c2fea0fbb16269a6f61d9555178808e2e8c7b3303046c89190d8ca9a3715d81" Oct 30 10:57:47.804628 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.804568 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c2fea0fbb16269a6f61d9555178808e2e8c7b3303046c89190d8ca9a3715d81"} err="failed to get container status \"5c2fea0fbb16269a6f61d9555178808e2e8c7b3303046c89190d8ca9a3715d81\": rpc error: code = NotFound desc = could not find container \"5c2fea0fbb16269a6f61d9555178808e2e8c7b3303046c89190d8ca9a3715d81\": container with ID starting with 5c2fea0fbb16269a6f61d9555178808e2e8c7b3303046c89190d8ca9a3715d81 not found: ID does not exist" Oct 30 10:57:47.844000 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.843973 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:47.855863 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.855838 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:47.885833 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.885807 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:47.886227 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.886214 4289 cpu_manager.go:414] "containerMap: removing container" podUID="5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f" containerName="korrel8r" Oct 30 10:57:47.886269 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.886229 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f" containerName="korrel8r" Oct 30 10:57:47.886269 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.886258 4289 cpu_manager.go:414] "containerMap: removing container" podUID="5c814605-a67a-42fa-8e9c-af9c352ff482" containerName="otc-container" Oct 30 10:57:47.886269 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.886263 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c814605-a67a-42fa-8e9c-af9c352ff482" containerName="otc-container" Oct 30 10:57:47.886364 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.886333 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c814605-a67a-42fa-8e9c-af9c352ff482" containerName="otc-container" Oct 30 10:57:47.886364 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.886345 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d6b4edf-57b0-4ac9-848d-d7abd5f9ed0f" containerName="korrel8r" Oct 30 10:57:47.890637 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.890622 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:47.906743 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.906722 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 10:57:47.906869 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.906848 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-e4cadd38\"" Oct 30 10:57:47.906989 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.906935 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 10:57:47.907057 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.907028 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 10:57:47.907095 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.907056 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8vrjb\"" Oct 30 10:57:47.908592 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:47.908576 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:48.041620 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:48.041527 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:48.041620 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:48.041587 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dnc7\" (UniqueName: \"kubernetes.io/projected/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-kube-api-access-4dnc7\") pod \"data-science-collector-collector-1\" (UID: \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:48.041814 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:48.041708 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:48.041814 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:48.041747 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:48.041814 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:48.041794 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:48.142637 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:48.142601 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:48.142637 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:48.142650 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:48.142861 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:48.142702 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:48.142861 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:48.142739 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:48.142861 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:48.142784 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dnc7\" (UniqueName: \"kubernetes.io/projected/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-kube-api-access-4dnc7\") pod \"data-science-collector-collector-1\" (UID: \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:48.143505 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:48.143475 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:48.145052 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:48.145027 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:48.145151 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:48.145116 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:48.145151 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:48.145118 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:48.165695 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:48.165665 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dnc7\" (UniqueName: \"kubernetes.io/projected/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-kube-api-access-4dnc7\") pod \"data-science-collector-collector-1\" (UID: \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:48.203940 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:48.203910 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:48.270730 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:48.270699 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:48.497755 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:48.497731 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:48.498638 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:57:48.498614 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode640f1e2_f5d3_4368_9916_5e4c1b9cfd89.slice/crio-8b33e9611d7a8dee00af401a6148e0c8e8a68ebc7a3e81d2926665d764d1559b WatchSource:0}: Error finding container 8b33e9611d7a8dee00af401a6148e0c8e8a68ebc7a3e81d2926665d764d1559b: Status 404 returned error can't find the container with id 8b33e9611d7a8dee00af401a6148e0c8e8a68ebc7a3e81d2926665d764d1559b Oct 30 10:57:48.501143 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:48.501113 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:57:48.501255 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:48.501165 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:57:48.803117 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:48.803039 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89","Type":"ContainerStarted","Data":"eafb3495a617fb0dd2567ac30b1dc8f92500a3b82d234f25fa7751e5d410a55d"} Oct 30 10:57:48.803117 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:48.803087 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89","Type":"ContainerStarted","Data":"8b33e9611d7a8dee00af401a6148e0c8e8a68ebc7a3e81d2926665d764d1559b"} Oct 30 10:57:48.847861 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:48.847813 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.847799489 podStartE2EDuration="1.847799489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:57:48.846362411 +0000 UTC m=+759.347741897" watchObservedRunningTime="2026-10-30 10:57:48.847799489 +0000 UTC m=+759.349178974" Oct 30 10:57:49.624203 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:49.624170 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c814605-a67a-42fa-8e9c-af9c352ff482" path="/var/lib/kubelet/pods/5c814605-a67a-42fa-8e9c-af9c352ff482/volumes" Oct 30 10:57:54.077359 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:54.077267 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:54.277656 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:54.277615 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:54.443589 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:54.443499 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:54.801010 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:54.800963 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:55.762745 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:55.762711 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:55.763151 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:55.762897 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="e640f1e2-f5d3-4368-9916-5e4c1b9cfd89" containerName="otc-container" containerID="cri-o://eafb3495a617fb0dd2567ac30b1dc8f92500a3b82d234f25fa7751e5d410a55d" gracePeriod=30 Oct 30 10:57:55.929209 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:55.929174 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:56.013732 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.013674 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:56.094752 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.094718 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:57:56.110817 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.110787 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-monitoring-tls-certs\") pod \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\" (UID: \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\") " Oct 30 10:57:56.110948 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.110854 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-otc-internal\" (UniqueName: \"kubernetes.io/configmap/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-otc-internal\") pod \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\" (UID: \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\") " Oct 30 10:57:56.110996 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.110961 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-data-science-collector-ta-client-cert\") pod \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\" (UID: \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\") " Oct 30 10:57:56.111049 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.111032 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-kube-api-access-4dnc7\" (UniqueName: \"kubernetes.io/projected/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-kube-api-access-4dnc7\") pod \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\" (UID: \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\") " Oct 30 10:57:56.111104 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.111079 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-tls-certs\" (UniqueName: \"kubernetes.io/secret/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-tls-certs\") pod \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\" (UID: \"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89\") " Oct 30 10:57:56.111223 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.111169 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-otc-internal" pod "e640f1e2-f5d3-4368-9916-5e4c1b9cfd89" (UID: "e640f1e2-f5d3-4368-9916-5e4c1b9cfd89"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:57:56.111400 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.111364 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:56.113070 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.113040 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-tls-certs" pod "e640f1e2-f5d3-4368-9916-5e4c1b9cfd89" (UID: "e640f1e2-f5d3-4368-9916-5e4c1b9cfd89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:57:56.113168 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.113086 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-data-science-collector-ta-client-cert" pod "e640f1e2-f5d3-4368-9916-5e4c1b9cfd89" (UID: "e640f1e2-f5d3-4368-9916-5e4c1b9cfd89"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:57:56.113168 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.113123 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-monitoring-tls-certs" pod "e640f1e2-f5d3-4368-9916-5e4c1b9cfd89" (UID: "e640f1e2-f5d3-4368-9916-5e4c1b9cfd89"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:57:56.113168 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.113139 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-kube-api-access-4dnc7" pod "e640f1e2-f5d3-4368-9916-5e4c1b9cfd89" (UID: "e640f1e2-f5d3-4368-9916-5e4c1b9cfd89"). InnerVolumeSpecName "kube-api-access-4dnc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:57:56.212754 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.212717 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dnc7\" (UniqueName: \"kubernetes.io/projected/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-kube-api-access-4dnc7\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:56.212754 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.212746 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:56.212754 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.212755 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:56.212974 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.212766 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:57:56.257607 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:57:56.257569 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:57:56.840759 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.840720 4289 generic.go:356] "Generic (PLEG): container finished" podID="e640f1e2-f5d3-4368-9916-5e4c1b9cfd89" containerID="eafb3495a617fb0dd2567ac30b1dc8f92500a3b82d234f25fa7751e5d410a55d" exitCode=0 Oct 30 10:57:56.840759 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.840751 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89","Type":"ContainerDied","Data":"eafb3495a617fb0dd2567ac30b1dc8f92500a3b82d234f25fa7751e5d410a55d"} Oct 30 10:57:56.841311 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.840774 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:56.841311 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.840812 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e640f1e2-f5d3-4368-9916-5e4c1b9cfd89","Type":"ContainerDied","Data":"8b33e9611d7a8dee00af401a6148e0c8e8a68ebc7a3e81d2926665d764d1559b"} Oct 30 10:57:56.841311 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.840835 4289 scope.go:122] "RemoveContainer" containerID="eafb3495a617fb0dd2567ac30b1dc8f92500a3b82d234f25fa7751e5d410a55d" Oct 30 10:57:56.850128 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.850112 4289 scope.go:122] "RemoveContainer" containerID="eafb3495a617fb0dd2567ac30b1dc8f92500a3b82d234f25fa7751e5d410a55d" Oct 30 10:57:56.850395 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:57:56.850378 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eafb3495a617fb0dd2567ac30b1dc8f92500a3b82d234f25fa7751e5d410a55d\": container with ID starting with eafb3495a617fb0dd2567ac30b1dc8f92500a3b82d234f25fa7751e5d410a55d not found: ID does not exist" containerID="eafb3495a617fb0dd2567ac30b1dc8f92500a3b82d234f25fa7751e5d410a55d" Oct 30 10:57:56.850536 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.850405 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eafb3495a617fb0dd2567ac30b1dc8f92500a3b82d234f25fa7751e5d410a55d"} err="failed to get container status \"eafb3495a617fb0dd2567ac30b1dc8f92500a3b82d234f25fa7751e5d410a55d\": rpc error: code = NotFound desc = could not find container \"eafb3495a617fb0dd2567ac30b1dc8f92500a3b82d234f25fa7751e5d410a55d\": container with ID starting with eafb3495a617fb0dd2567ac30b1dc8f92500a3b82d234f25fa7751e5d410a55d not found: ID does not exist" Oct 30 10:57:56.889724 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.889698 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:56.893411 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.893385 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:56.930927 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.930897 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:56.931293 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.931281 4289 cpu_manager.go:414] "containerMap: removing container" podUID="e640f1e2-f5d3-4368-9916-5e4c1b9cfd89" containerName="otc-container" Oct 30 10:57:56.931349 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.931295 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e640f1e2-f5d3-4368-9916-5e4c1b9cfd89" containerName="otc-container" Oct 30 10:57:56.931388 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.931360 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="e640f1e2-f5d3-4368-9916-5e4c1b9cfd89" containerName="otc-container" Oct 30 10:57:56.935750 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.935733 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:56.952137 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.952114 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 10:57:56.952298 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.952179 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 10:57:56.952298 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.952236 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-e4cadd38\"" Oct 30 10:57:56.952298 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.952271 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8vrjb\"" Oct 30 10:57:56.952499 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.952488 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 10:57:56.954221 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:56.954200 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:57.020074 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:57.020045 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:57.020074 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:57.020078 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:57.020256 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:57.020108 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:57.020256 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:57.020229 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t9td\" (UniqueName: \"kubernetes.io/projected/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-kube-api-access-7t9td\") pod \"data-science-collector-collector-1\" (UID: \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:57.020352 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:57.020262 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:57.121391 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:57.121287 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:57.121391 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:57.121348 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:57.121650 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:57.121412 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t9td\" (UniqueName: \"kubernetes.io/projected/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-kube-api-access-7t9td\") pod \"data-science-collector-collector-1\" (UID: \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:57.121650 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:57.121462 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:57.121650 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:57.121553 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:57.122212 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:57.122188 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:57.123997 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:57.123967 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:57.123997 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:57.123978 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:57.123997 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:57.123972 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:57.143507 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:57.143481 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t9td\" (UniqueName: \"kubernetes.io/projected/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-kube-api-access-7t9td\") pod \"data-science-collector-collector-1\" (UID: \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:57.248637 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:57.248600 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:57:57.542859 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:57:57.542826 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4ae1f4_0b8d_45d6_be89_eac93cbabe26.slice/crio-00ce9c2eee305505dd028dcb7a5bdca0d545b40bfc8488cba9d15bb76e047d13 WatchSource:0}: Error finding container 00ce9c2eee305505dd028dcb7a5bdca0d545b40bfc8488cba9d15bb76e047d13: Status 404 returned error can't find the container with id 00ce9c2eee305505dd028dcb7a5bdca0d545b40bfc8488cba9d15bb76e047d13 Oct 30 10:57:57.542958 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:57.542876 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:57:57.544962 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:57.544935 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:57:57.545045 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:57.544990 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:57:57.631740 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:57.631701 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e640f1e2-f5d3-4368-9916-5e4c1b9cfd89" path="/var/lib/kubelet/pods/e640f1e2-f5d3-4368-9916-5e4c1b9cfd89/volumes" Oct 30 10:57:57.848311 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:57.848227 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26","Type":"ContainerStarted","Data":"7cde8198fd49eaf9893d9af8c641331f9f9569c23d13069678d1d98ab62da6db"} Oct 30 10:57:57.848311 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:57.848292 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26","Type":"ContainerStarted","Data":"00ce9c2eee305505dd028dcb7a5bdca0d545b40bfc8488cba9d15bb76e047d13"} Oct 30 10:57:57.890324 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:57:57.890273 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.890258575 podStartE2EDuration="1.890258575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:57:57.889051618 +0000 UTC m=+768.390431123" watchObservedRunningTime="2026-10-30 10:57:57.890258575 +0000 UTC m=+768.391638059" Oct 30 10:58:02.125346 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:02.125298 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:02.328526 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:02.328478 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:02.489930 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:02.489893 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:02.655552 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:02.655522 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:03.822556 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:03.822521 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:58:03.822974 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:03.822725 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="8d4ae1f4-0b8d-45d6-be89-eac93cbabe26" containerName="otc-container" containerID="cri-o://7cde8198fd49eaf9893d9af8c641331f9f9569c23d13069678d1d98ab62da6db" gracePeriod=30 Oct 30 10:58:03.997340 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:03.997297 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:04.164872 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:04.164757 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:05.064177 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.064153 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:05.198756 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.198670 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-kube-api-access-7t9td\" (UniqueName: \"kubernetes.io/projected/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-kube-api-access-7t9td\") pod \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\" (UID: \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\") " Oct 30 10:58:05.198756 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.198744 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-otc-internal\" (UniqueName: \"kubernetes.io/configmap/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-otc-internal\") pod \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\" (UID: \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\") " Oct 30 10:58:05.198978 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.198789 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-monitoring-tls-certs\") pod \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\" (UID: \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\") " Oct 30 10:58:05.198978 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.198817 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-tls-certs\" (UniqueName: \"kubernetes.io/secret/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-tls-certs\") pod \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\" (UID: \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\") " Oct 30 10:58:05.198978 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.198903 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-data-science-collector-ta-client-cert\") pod \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\" (UID: \"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26\") " Oct 30 10:58:05.199117 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.199066 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-otc-internal" pod "8d4ae1f4-0b8d-45d6-be89-eac93cbabe26" (UID: "8d4ae1f4-0b8d-45d6-be89-eac93cbabe26"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:58:05.199227 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.199207 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:58:05.200961 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.200928 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-monitoring-tls-certs" pod "8d4ae1f4-0b8d-45d6-be89-eac93cbabe26" (UID: "8d4ae1f4-0b8d-45d6-be89-eac93cbabe26"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:58:05.201089 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.200953 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-tls-certs" pod "8d4ae1f4-0b8d-45d6-be89-eac93cbabe26" (UID: "8d4ae1f4-0b8d-45d6-be89-eac93cbabe26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:58:05.201089 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.201047 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-kube-api-access-7t9td" pod "8d4ae1f4-0b8d-45d6-be89-eac93cbabe26" (UID: "8d4ae1f4-0b8d-45d6-be89-eac93cbabe26"). InnerVolumeSpecName "kube-api-access-7t9td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:58:05.201089 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.201056 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-data-science-collector-ta-client-cert" pod "8d4ae1f4-0b8d-45d6-be89-eac93cbabe26" (UID: "8d4ae1f4-0b8d-45d6-be89-eac93cbabe26"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:58:05.299856 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.299820 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:58:05.299856 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.299848 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t9td\" (UniqueName: \"kubernetes.io/projected/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-kube-api-access-7t9td\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:58:05.299856 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.299861 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:58:05.300071 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.299875 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:58:05.886904 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.886825 4289 generic.go:356] "Generic (PLEG): container finished" podID="8d4ae1f4-0b8d-45d6-be89-eac93cbabe26" containerID="7cde8198fd49eaf9893d9af8c641331f9f9569c23d13069678d1d98ab62da6db" exitCode=0 Oct 30 10:58:05.886904 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.886889 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26","Type":"ContainerDied","Data":"7cde8198fd49eaf9893d9af8c641331f9f9569c23d13069678d1d98ab62da6db"} Oct 30 10:58:05.887112 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.886942 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:05.887112 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.886955 4289 scope.go:122] "RemoveContainer" containerID="7cde8198fd49eaf9893d9af8c641331f9f9569c23d13069678d1d98ab62da6db" Oct 30 10:58:05.887112 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.886945 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8d4ae1f4-0b8d-45d6-be89-eac93cbabe26","Type":"ContainerDied","Data":"00ce9c2eee305505dd028dcb7a5bdca0d545b40bfc8488cba9d15bb76e047d13"} Oct 30 10:58:05.895481 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.895459 4289 scope.go:122] "RemoveContainer" containerID="7cde8198fd49eaf9893d9af8c641331f9f9569c23d13069678d1d98ab62da6db" Oct 30 10:58:05.895740 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:58:05.895721 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cde8198fd49eaf9893d9af8c641331f9f9569c23d13069678d1d98ab62da6db\": container with ID starting with 7cde8198fd49eaf9893d9af8c641331f9f9569c23d13069678d1d98ab62da6db not found: ID does not exist" containerID="7cde8198fd49eaf9893d9af8c641331f9f9569c23d13069678d1d98ab62da6db" Oct 30 10:58:05.895794 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.895748 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cde8198fd49eaf9893d9af8c641331f9f9569c23d13069678d1d98ab62da6db"} err="failed to get container status \"7cde8198fd49eaf9893d9af8c641331f9f9569c23d13069678d1d98ab62da6db\": rpc error: code = NotFound desc = could not find container \"7cde8198fd49eaf9893d9af8c641331f9f9569c23d13069678d1d98ab62da6db\": container with ID starting with 7cde8198fd49eaf9893d9af8c641331f9f9569c23d13069678d1d98ab62da6db not found: ID does not exist" Oct 30 10:58:05.935751 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.935726 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:58:05.947606 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.947581 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:58:05.975750 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.975726 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:58:05.976126 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.976112 4289 cpu_manager.go:414] "containerMap: removing container" podUID="8d4ae1f4-0b8d-45d6-be89-eac93cbabe26" containerName="otc-container" Oct 30 10:58:05.976175 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.976127 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d4ae1f4-0b8d-45d6-be89-eac93cbabe26" containerName="otc-container" Oct 30 10:58:05.976209 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.976194 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d4ae1f4-0b8d-45d6-be89-eac93cbabe26" containerName="otc-container" Oct 30 10:58:05.980714 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.980699 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:05.997122 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.997102 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 10:58:05.997122 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.997102 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 10:58:05.997313 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.997148 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 10:58:05.997392 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.997376 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-e4cadd38\"" Oct 30 10:58:05.997558 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.997540 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8vrjb\"" Oct 30 10:58:05.999136 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:05.999114 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:58:06.107866 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:06.107829 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:06.107866 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:06.107863 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:06.108267 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:06.107881 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:06.108267 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:06.108040 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:06.108267 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:06.108117 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4hsd\" (UniqueName: \"kubernetes.io/projected/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-kube-api-access-r4hsd\") pod \"data-science-collector-collector-1\" (UID: \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:06.208832 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:06.208742 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:06.208832 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:06.208784 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:06.208832 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:06.208809 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:06.209107 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:06.208844 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:06.209107 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:06.208892 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4hsd\" (UniqueName: \"kubernetes.io/projected/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-kube-api-access-r4hsd\") pod \"data-science-collector-collector-1\" (UID: \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:06.209643 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:06.209614 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:06.211344 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:06.211315 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:06.211344 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:06.211331 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:06.211494 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:06.211405 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:06.231193 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:06.231168 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4hsd\" (UniqueName: \"kubernetes.io/projected/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-kube-api-access-r4hsd\") pod \"data-science-collector-collector-1\" (UID: \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:06.293169 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:06.293139 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:06.585475 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:06.585451 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:58:06.586868 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:58:06.586842 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6e4ce1_6989_42e8_9672_336d85bd7ed6.slice/crio-f11cc5dfd5c4fb1d33e3dbe9124b9d5b66303949be9b9c555b91584368c13330 WatchSource:0}: Error finding container f11cc5dfd5c4fb1d33e3dbe9124b9d5b66303949be9b9c555b91584368c13330: Status 404 returned error can't find the container with id f11cc5dfd5c4fb1d33e3dbe9124b9d5b66303949be9b9c555b91584368c13330 Oct 30 10:58:06.588959 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:06.588926 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:58:06.589014 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:06.588990 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:58:06.893424 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:06.893337 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ba6e4ce1-6989-42e8-9672-336d85bd7ed6","Type":"ContainerStarted","Data":"4ff089826b31f59388f7a031a343d0d4f5e6c38d234a4b6bf89e6b41b4be9082"} Oct 30 10:58:06.893424 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:06.893398 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ba6e4ce1-6989-42e8-9672-336d85bd7ed6","Type":"ContainerStarted","Data":"f11cc5dfd5c4fb1d33e3dbe9124b9d5b66303949be9b9c555b91584368c13330"} Oct 30 10:58:06.935203 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:06.935154 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.9351411189999999 podStartE2EDuration="1.935141119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:58:06.935099102 +0000 UTC m=+777.436478586" watchObservedRunningTime="2026-10-30 10:58:06.935141119 +0000 UTC m=+777.436520603" Oct 30 10:58:07.624899 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:07.624864 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d4ae1f4-0b8d-45d6-be89-eac93cbabe26" path="/var/lib/kubelet/pods/8d4ae1f4-0b8d-45d6-be89-eac93cbabe26/volumes" Oct 30 10:58:10.074099 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:10.074062 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:10.255304 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:10.255265 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:11.852607 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:11.852567 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:12.016160 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:12.016109 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:16.906700 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:16.906667 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:58:16.907560 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:16.907527 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="ba6e4ce1-6989-42e8-9672-336d85bd7ed6" containerName="otc-container" containerID="cri-o://4ff089826b31f59388f7a031a343d0d4f5e6c38d234a4b6bf89e6b41b4be9082" gracePeriod=30 Oct 30 10:58:17.900215 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:17.900175 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:18.099135 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.099094 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:18.153199 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.153138 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:18.291729 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.291690 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:18.319486 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.319455 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-monitoring-tls-certs\") pod \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\" (UID: \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\") " Oct 30 10:58:18.319627 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.319550 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-otc-internal\") pod \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\" (UID: \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\") " Oct 30 10:58:18.319627 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.319609 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-tls-certs\") pod \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\" (UID: \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\") " Oct 30 10:58:18.319705 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.319647 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-data-science-collector-ta-client-cert\") pod \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\" (UID: \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\") " Oct 30 10:58:18.319744 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.319718 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-kube-api-access-r4hsd\" (UniqueName: \"kubernetes.io/projected/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-kube-api-access-r4hsd\") pod \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\" (UID: \"ba6e4ce1-6989-42e8-9672-336d85bd7ed6\") " Oct 30 10:58:18.319945 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.319909 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-otc-internal" pod "ba6e4ce1-6989-42e8-9672-336d85bd7ed6" (UID: "ba6e4ce1-6989-42e8-9672-336d85bd7ed6"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:58:18.320077 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.319999 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:58:18.321773 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.321742 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-data-science-collector-ta-client-cert" pod "ba6e4ce1-6989-42e8-9672-336d85bd7ed6" (UID: "ba6e4ce1-6989-42e8-9672-336d85bd7ed6"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:58:18.321957 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.321803 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-monitoring-tls-certs" pod "ba6e4ce1-6989-42e8-9672-336d85bd7ed6" (UID: "ba6e4ce1-6989-42e8-9672-336d85bd7ed6"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:58:18.321957 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.321813 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-tls-certs" pod "ba6e4ce1-6989-42e8-9672-336d85bd7ed6" (UID: "ba6e4ce1-6989-42e8-9672-336d85bd7ed6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:58:18.322190 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.322173 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-kube-api-access-r4hsd" pod "ba6e4ce1-6989-42e8-9672-336d85bd7ed6" (UID: "ba6e4ce1-6989-42e8-9672-336d85bd7ed6"). InnerVolumeSpecName "kube-api-access-r4hsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:58:18.420491 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.420401 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:58:18.420491 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.420448 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:58:18.420491 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.420461 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4hsd\" (UniqueName: \"kubernetes.io/projected/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-kube-api-access-r4hsd\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:58:18.420491 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.420472 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba6e4ce1-6989-42e8-9672-336d85bd7ed6-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:58:18.951101 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.951064 4289 generic.go:356] "Generic (PLEG): container finished" podID="ba6e4ce1-6989-42e8-9672-336d85bd7ed6" containerID="4ff089826b31f59388f7a031a343d0d4f5e6c38d234a4b6bf89e6b41b4be9082" exitCode=0 Oct 30 10:58:18.951292 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.951156 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ba6e4ce1-6989-42e8-9672-336d85bd7ed6","Type":"ContainerDied","Data":"4ff089826b31f59388f7a031a343d0d4f5e6c38d234a4b6bf89e6b41b4be9082"} Oct 30 10:58:18.951292 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.951182 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:18.951292 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.951215 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ba6e4ce1-6989-42e8-9672-336d85bd7ed6","Type":"ContainerDied","Data":"f11cc5dfd5c4fb1d33e3dbe9124b9d5b66303949be9b9c555b91584368c13330"} Oct 30 10:58:18.951292 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.951251 4289 scope.go:122] "RemoveContainer" containerID="4ff089826b31f59388f7a031a343d0d4f5e6c38d234a4b6bf89e6b41b4be9082" Oct 30 10:58:18.960612 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.960589 4289 scope.go:122] "RemoveContainer" containerID="4ff089826b31f59388f7a031a343d0d4f5e6c38d234a4b6bf89e6b41b4be9082" Oct 30 10:58:18.960949 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:58:18.960918 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ff089826b31f59388f7a031a343d0d4f5e6c38d234a4b6bf89e6b41b4be9082\": container with ID starting with 4ff089826b31f59388f7a031a343d0d4f5e6c38d234a4b6bf89e6b41b4be9082 not found: ID does not exist" containerID="4ff089826b31f59388f7a031a343d0d4f5e6c38d234a4b6bf89e6b41b4be9082" Oct 30 10:58:18.961034 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.960959 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ff089826b31f59388f7a031a343d0d4f5e6c38d234a4b6bf89e6b41b4be9082"} err="failed to get container status \"4ff089826b31f59388f7a031a343d0d4f5e6c38d234a4b6bf89e6b41b4be9082\": rpc error: code = NotFound desc = could not find container \"4ff089826b31f59388f7a031a343d0d4f5e6c38d234a4b6bf89e6b41b4be9082\": container with ID starting with 4ff089826b31f59388f7a031a343d0d4f5e6c38d234a4b6bf89e6b41b4be9082 not found: ID does not exist" Oct 30 10:58:18.999324 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:18.999300 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:58:19.011221 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.011189 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:58:19.040456 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.040419 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:58:19.040876 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.040862 4289 cpu_manager.go:414] "containerMap: removing container" podUID="ba6e4ce1-6989-42e8-9672-336d85bd7ed6" containerName="otc-container" Oct 30 10:58:19.040925 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.040878 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6e4ce1-6989-42e8-9672-336d85bd7ed6" containerName="otc-container" Oct 30 10:58:19.040959 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.040951 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6e4ce1-6989-42e8-9672-336d85bd7ed6" containerName="otc-container" Oct 30 10:58:19.045622 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.045605 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:19.063797 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.063780 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-e4cadd38\"" Oct 30 10:58:19.063924 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.063783 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 10:58:19.064014 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.063791 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 10:58:19.064014 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.063938 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8vrjb\"" Oct 30 10:58:19.064014 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.063880 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 10:58:19.065804 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.065783 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:58:19.125878 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.125854 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/01ae6b33-820a-472b-b244-a31dd8a0b460-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"01ae6b33-820a-472b-b244-a31dd8a0b460\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:19.126233 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.125911 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01ae6b33-820a-472b-b244-a31dd8a0b460-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"01ae6b33-820a-472b-b244-a31dd8a0b460\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:19.126233 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.125927 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbf7w\" (UniqueName: \"kubernetes.io/projected/01ae6b33-820a-472b-b244-a31dd8a0b460-kube-api-access-gbf7w\") pod \"data-science-collector-collector-1\" (UID: \"01ae6b33-820a-472b-b244-a31dd8a0b460\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:19.126233 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.126026 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/01ae6b33-820a-472b-b244-a31dd8a0b460-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"01ae6b33-820a-472b-b244-a31dd8a0b460\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:19.126233 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.126069 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/01ae6b33-820a-472b-b244-a31dd8a0b460-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"01ae6b33-820a-472b-b244-a31dd8a0b460\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:19.227262 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.227235 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01ae6b33-820a-472b-b244-a31dd8a0b460-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"01ae6b33-820a-472b-b244-a31dd8a0b460\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:19.227401 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.227270 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbf7w\" (UniqueName: \"kubernetes.io/projected/01ae6b33-820a-472b-b244-a31dd8a0b460-kube-api-access-gbf7w\") pod \"data-science-collector-collector-1\" (UID: \"01ae6b33-820a-472b-b244-a31dd8a0b460\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:19.227401 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.227306 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/01ae6b33-820a-472b-b244-a31dd8a0b460-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"01ae6b33-820a-472b-b244-a31dd8a0b460\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:19.227401 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.227356 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/01ae6b33-820a-472b-b244-a31dd8a0b460-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"01ae6b33-820a-472b-b244-a31dd8a0b460\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:19.227556 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.227417 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/01ae6b33-820a-472b-b244-a31dd8a0b460-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"01ae6b33-820a-472b-b244-a31dd8a0b460\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:19.228088 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.228067 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/01ae6b33-820a-472b-b244-a31dd8a0b460-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"01ae6b33-820a-472b-b244-a31dd8a0b460\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:19.229883 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.229860 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/01ae6b33-820a-472b-b244-a31dd8a0b460-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"01ae6b33-820a-472b-b244-a31dd8a0b460\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:19.230004 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.229924 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01ae6b33-820a-472b-b244-a31dd8a0b460-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"01ae6b33-820a-472b-b244-a31dd8a0b460\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:19.230004 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.229949 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/01ae6b33-820a-472b-b244-a31dd8a0b460-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"01ae6b33-820a-472b-b244-a31dd8a0b460\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:19.249731 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.249708 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbf7w\" (UniqueName: \"kubernetes.io/projected/01ae6b33-820a-472b-b244-a31dd8a0b460-kube-api-access-gbf7w\") pod \"data-science-collector-collector-1\" (UID: \"01ae6b33-820a-472b-b244-a31dd8a0b460\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:19.357406 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.357375 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:19.576538 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.576453 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:58:19.625914 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.625883 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba6e4ce1-6989-42e8-9672-336d85bd7ed6" path="/var/lib/kubelet/pods/ba6e4ce1-6989-42e8-9672-336d85bd7ed6/volumes" Oct 30 10:58:19.652945 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.652910 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:58:19.653078 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:58:19.653051 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ae6b33_820a_472b_b244_a31dd8a0b460.slice/crio-5e5210d26d0847273231782f6e2b4ce83d825d046e27581c3fef8298d5d17fe7 WatchSource:0}: Error finding container 5e5210d26d0847273231782f6e2b4ce83d825d046e27581c3fef8298d5d17fe7: Status 404 returned error can't find the container with id 5e5210d26d0847273231782f6e2b4ce83d825d046e27581c3fef8298d5d17fe7 Oct 30 10:58:19.655160 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.655129 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:58:19.655221 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.655187 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:58:19.749309 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.749278 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:19.915757 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.915664 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:19.958745 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.958714 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"01ae6b33-820a-472b-b244-a31dd8a0b460","Type":"ContainerStarted","Data":"39e12408f3e0723c072a6f4a4c66a62bae72ada55b59a8eab98dd4270af71ab0"} Oct 30 10:58:19.958907 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.958759 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"01ae6b33-820a-472b-b244-a31dd8a0b460","Type":"ContainerStarted","Data":"5e5210d26d0847273231782f6e2b4ce83d825d046e27581c3fef8298d5d17fe7"} Oct 30 10:58:19.958907 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:19.958767 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="01ae6b33-820a-472b-b244-a31dd8a0b460" containerName="otc-container" containerID="cri-o://39e12408f3e0723c072a6f4a4c66a62bae72ada55b59a8eab98dd4270af71ab0" gracePeriod=30 Oct 30 10:58:20.000990 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:20.000945 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.000930959 podStartE2EDuration="1.000930959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:58:19.999805813 +0000 UTC m=+790.501185297" watchObservedRunningTime="2026-10-30 10:58:20.000930959 +0000 UTC m=+790.502310443" Oct 30 10:58:20.191927 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:20.191900 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:20.336466 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:20.336413 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01ae6b33-820a-472b-b244-a31dd8a0b460-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/01ae6b33-820a-472b-b244-a31dd8a0b460-data-science-collector-ta-client-cert\") pod \"01ae6b33-820a-472b-b244-a31dd8a0b460\" (UID: \"01ae6b33-820a-472b-b244-a31dd8a0b460\") " Oct 30 10:58:20.336626 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:20.336477 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ae6b33-820a-472b-b244-a31dd8a0b460-kube-api-access-gbf7w\" (UniqueName: \"kubernetes.io/projected/01ae6b33-820a-472b-b244-a31dd8a0b460-kube-api-access-gbf7w\") pod \"01ae6b33-820a-472b-b244-a31dd8a0b460\" (UID: \"01ae6b33-820a-472b-b244-a31dd8a0b460\") " Oct 30 10:58:20.336626 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:20.336503 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01ae6b33-820a-472b-b244-a31dd8a0b460-otc-internal\" (UniqueName: \"kubernetes.io/configmap/01ae6b33-820a-472b-b244-a31dd8a0b460-otc-internal\") pod \"01ae6b33-820a-472b-b244-a31dd8a0b460\" (UID: \"01ae6b33-820a-472b-b244-a31dd8a0b460\") " Oct 30 10:58:20.336626 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:20.336562 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01ae6b33-820a-472b-b244-a31dd8a0b460-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/01ae6b33-820a-472b-b244-a31dd8a0b460-monitoring-tls-certs\") pod \"01ae6b33-820a-472b-b244-a31dd8a0b460\" (UID: \"01ae6b33-820a-472b-b244-a31dd8a0b460\") " Oct 30 10:58:20.336792 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:20.336616 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01ae6b33-820a-472b-b244-a31dd8a0b460-tls-certs\" (UniqueName: \"kubernetes.io/secret/01ae6b33-820a-472b-b244-a31dd8a0b460-tls-certs\") pod \"01ae6b33-820a-472b-b244-a31dd8a0b460\" (UID: \"01ae6b33-820a-472b-b244-a31dd8a0b460\") " Oct 30 10:58:20.336908 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:20.336881 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ae6b33-820a-472b-b244-a31dd8a0b460-otc-internal" pod "01ae6b33-820a-472b-b244-a31dd8a0b460" (UID: "01ae6b33-820a-472b-b244-a31dd8a0b460"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:58:20.338653 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:20.338624 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ae6b33-820a-472b-b244-a31dd8a0b460-tls-certs" pod "01ae6b33-820a-472b-b244-a31dd8a0b460" (UID: "01ae6b33-820a-472b-b244-a31dd8a0b460"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:58:20.338750 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:20.338634 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ae6b33-820a-472b-b244-a31dd8a0b460-kube-api-access-gbf7w" pod "01ae6b33-820a-472b-b244-a31dd8a0b460" (UID: "01ae6b33-820a-472b-b244-a31dd8a0b460"). InnerVolumeSpecName "kube-api-access-gbf7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:58:20.338980 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:20.338965 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ae6b33-820a-472b-b244-a31dd8a0b460-data-science-collector-ta-client-cert" pod "01ae6b33-820a-472b-b244-a31dd8a0b460" (UID: "01ae6b33-820a-472b-b244-a31dd8a0b460"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:58:20.339044 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:20.338986 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ae6b33-820a-472b-b244-a31dd8a0b460-monitoring-tls-certs" pod "01ae6b33-820a-472b-b244-a31dd8a0b460" (UID: "01ae6b33-820a-472b-b244-a31dd8a0b460"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:58:20.437886 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:20.437806 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/01ae6b33-820a-472b-b244-a31dd8a0b460-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:58:20.437886 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:20.437834 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01ae6b33-820a-472b-b244-a31dd8a0b460-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:58:20.437886 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:20.437845 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/01ae6b33-820a-472b-b244-a31dd8a0b460-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:58:20.437886 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:20.437855 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbf7w\" (UniqueName: \"kubernetes.io/projected/01ae6b33-820a-472b-b244-a31dd8a0b460-kube-api-access-gbf7w\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:58:20.437886 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:20.437864 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/01ae6b33-820a-472b-b244-a31dd8a0b460-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:58:20.964617 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:20.964580 4289 generic.go:356] "Generic (PLEG): container finished" podID="01ae6b33-820a-472b-b244-a31dd8a0b460" containerID="39e12408f3e0723c072a6f4a4c66a62bae72ada55b59a8eab98dd4270af71ab0" exitCode=0 Oct 30 10:58:20.964813 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:20.964629 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:20.964813 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:20.964660 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"01ae6b33-820a-472b-b244-a31dd8a0b460","Type":"ContainerDied","Data":"39e12408f3e0723c072a6f4a4c66a62bae72ada55b59a8eab98dd4270af71ab0"} Oct 30 10:58:20.964813 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:20.964723 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"01ae6b33-820a-472b-b244-a31dd8a0b460","Type":"ContainerDied","Data":"5e5210d26d0847273231782f6e2b4ce83d825d046e27581c3fef8298d5d17fe7"} Oct 30 10:58:20.964813 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:20.964757 4289 scope.go:122] "RemoveContainer" containerID="39e12408f3e0723c072a6f4a4c66a62bae72ada55b59a8eab98dd4270af71ab0" Oct 30 10:58:20.973709 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:20.973691 4289 scope.go:122] "RemoveContainer" containerID="39e12408f3e0723c072a6f4a4c66a62bae72ada55b59a8eab98dd4270af71ab0" Oct 30 10:58:20.973970 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:58:20.973950 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39e12408f3e0723c072a6f4a4c66a62bae72ada55b59a8eab98dd4270af71ab0\": container with ID starting with 39e12408f3e0723c072a6f4a4c66a62bae72ada55b59a8eab98dd4270af71ab0 not found: ID does not exist" containerID="39e12408f3e0723c072a6f4a4c66a62bae72ada55b59a8eab98dd4270af71ab0" Oct 30 10:58:20.974029 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:20.973980 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39e12408f3e0723c072a6f4a4c66a62bae72ada55b59a8eab98dd4270af71ab0"} err="failed to get container status \"39e12408f3e0723c072a6f4a4c66a62bae72ada55b59a8eab98dd4270af71ab0\": rpc error: code = NotFound desc = could not find container \"39e12408f3e0723c072a6f4a4c66a62bae72ada55b59a8eab98dd4270af71ab0\": container with ID starting with 39e12408f3e0723c072a6f4a4c66a62bae72ada55b59a8eab98dd4270af71ab0 not found: ID does not exist" Oct 30 10:58:21.014685 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.014657 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:58:21.026565 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.026541 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:58:21.054168 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.054134 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:58:21.054628 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.054610 4289 cpu_manager.go:414] "containerMap: removing container" podUID="01ae6b33-820a-472b-b244-a31dd8a0b460" containerName="otc-container" Oct 30 10:58:21.054724 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.054629 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01ae6b33-820a-472b-b244-a31dd8a0b460" containerName="otc-container" Oct 30 10:58:21.054787 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.054747 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="01ae6b33-820a-472b-b244-a31dd8a0b460" containerName="otc-container" Oct 30 10:58:21.059136 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.059117 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:21.073779 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.073753 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-e4cadd38\"" Oct 30 10:58:21.074861 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.074839 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:58:21.143540 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.143503 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5366129e-7a00-4d73-9570-45788f871476-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5366129e-7a00-4d73-9570-45788f871476\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:21.143678 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.143614 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5366129e-7a00-4d73-9570-45788f871476-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5366129e-7a00-4d73-9570-45788f871476\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:21.143720 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.143688 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5366129e-7a00-4d73-9570-45788f871476-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5366129e-7a00-4d73-9570-45788f871476\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:21.143755 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.143717 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5366129e-7a00-4d73-9570-45788f871476-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5366129e-7a00-4d73-9570-45788f871476\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:21.143755 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.143742 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scdbs\" (UniqueName: \"kubernetes.io/projected/5366129e-7a00-4d73-9570-45788f871476-kube-api-access-scdbs\") pod \"data-science-collector-collector-1\" (UID: \"5366129e-7a00-4d73-9570-45788f871476\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:21.156498 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.156472 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8vrjb\"" Oct 30 10:58:21.156603 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.156531 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 10:58:21.156603 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.156587 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 10:58:21.156677 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.156667 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 10:58:21.245005 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.244978 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5366129e-7a00-4d73-9570-45788f871476-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5366129e-7a00-4d73-9570-45788f871476\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:21.245392 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.245026 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5366129e-7a00-4d73-9570-45788f871476-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5366129e-7a00-4d73-9570-45788f871476\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:21.245392 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.245049 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5366129e-7a00-4d73-9570-45788f871476-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5366129e-7a00-4d73-9570-45788f871476\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:21.245392 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.245079 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scdbs\" (UniqueName: \"kubernetes.io/projected/5366129e-7a00-4d73-9570-45788f871476-kube-api-access-scdbs\") pod \"data-science-collector-collector-1\" (UID: \"5366129e-7a00-4d73-9570-45788f871476\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:21.245392 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.245098 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5366129e-7a00-4d73-9570-45788f871476-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5366129e-7a00-4d73-9570-45788f871476\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:21.245811 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.245787 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5366129e-7a00-4d73-9570-45788f871476-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5366129e-7a00-4d73-9570-45788f871476\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:21.247392 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.247370 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5366129e-7a00-4d73-9570-45788f871476-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5366129e-7a00-4d73-9570-45788f871476\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:21.247499 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.247409 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5366129e-7a00-4d73-9570-45788f871476-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5366129e-7a00-4d73-9570-45788f871476\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:21.247588 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.247569 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5366129e-7a00-4d73-9570-45788f871476-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5366129e-7a00-4d73-9570-45788f871476\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:21.270218 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.270196 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scdbs\" (UniqueName: \"kubernetes.io/projected/5366129e-7a00-4d73-9570-45788f871476-kube-api-access-scdbs\") pod \"data-science-collector-collector-1\" (UID: \"5366129e-7a00-4d73-9570-45788f871476\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:21.371961 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.371927 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:21.625476 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.625368 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01ae6b33-820a-472b-b244-a31dd8a0b460" path="/var/lib/kubelet/pods/01ae6b33-820a-472b-b244-a31dd8a0b460/volumes" Oct 30 10:58:21.667437 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.667399 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:58:21.667966 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:58:21.667940 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5366129e_7a00_4d73_9570_45788f871476.slice/crio-f8be5879da54b637f108686ac22f7916444a16ecbb681b6710c817a59bf04509 WatchSource:0}: Error finding container f8be5879da54b637f108686ac22f7916444a16ecbb681b6710c817a59bf04509: Status 404 returned error can't find the container with id f8be5879da54b637f108686ac22f7916444a16ecbb681b6710c817a59bf04509 Oct 30 10:58:21.670140 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.670109 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:58:21.670210 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.670162 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:58:21.971711 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.971672 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5366129e-7a00-4d73-9570-45788f871476","Type":"ContainerStarted","Data":"2815f69c90d4e91012f46ac314e91f86d77ed421d117527d2f666ba590740561"} Oct 30 10:58:21.971890 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:21.971733 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5366129e-7a00-4d73-9570-45788f871476","Type":"ContainerStarted","Data":"f8be5879da54b637f108686ac22f7916444a16ecbb681b6710c817a59bf04509"} Oct 30 10:58:22.013346 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:22.013287 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.013274981 podStartE2EDuration="1.013274981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:58:22.011975077 +0000 UTC m=+792.513354562" watchObservedRunningTime="2026-10-30 10:58:22.013274981 +0000 UTC m=+792.514654465" Oct 30 10:58:22.943110 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:22.943074 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:23.109709 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:23.109672 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:23.647652 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:23.647615 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:26.257797 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:58:26.257757 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:58:29.150174 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:29.150130 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:29.350971 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:29.350936 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:29.519367 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:29.519330 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:29.862340 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:29.862258 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:30.029144 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:30.029112 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:30.834755 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:30.834721 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:58:30.835130 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:30.834890 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="5366129e-7a00-4d73-9570-45788f871476" containerName="otc-container" containerID="cri-o://2815f69c90d4e91012f46ac314e91f86d77ed421d117527d2f666ba590740561" gracePeriod=30 Oct 30 10:58:31.015315 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:31.015277 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:31.018151 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:31.018119 4289 generic.go:356] "Generic (PLEG): container finished" podID="5366129e-7a00-4d73-9570-45788f871476" containerID="2815f69c90d4e91012f46ac314e91f86d77ed421d117527d2f666ba590740561" exitCode=0 Oct 30 10:58:31.018295 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:31.018195 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5366129e-7a00-4d73-9570-45788f871476","Type":"ContainerDied","Data":"2815f69c90d4e91012f46ac314e91f86d77ed421d117527d2f666ba590740561"} Oct 30 10:58:31.074812 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:31.074787 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:31.132415 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:31.132337 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5366129e-7a00-4d73-9570-45788f871476-tls-certs\" (UniqueName: \"kubernetes.io/secret/5366129e-7a00-4d73-9570-45788f871476-tls-certs\") pod \"5366129e-7a00-4d73-9570-45788f871476\" (UID: \"5366129e-7a00-4d73-9570-45788f871476\") " Oct 30 10:58:31.132415 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:31.132374 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5366129e-7a00-4d73-9570-45788f871476-kube-api-access-scdbs\" (UniqueName: \"kubernetes.io/projected/5366129e-7a00-4d73-9570-45788f871476-kube-api-access-scdbs\") pod \"5366129e-7a00-4d73-9570-45788f871476\" (UID: \"5366129e-7a00-4d73-9570-45788f871476\") " Oct 30 10:58:31.132415 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:31.132398 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5366129e-7a00-4d73-9570-45788f871476-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5366129e-7a00-4d73-9570-45788f871476-otc-internal\") pod \"5366129e-7a00-4d73-9570-45788f871476\" (UID: \"5366129e-7a00-4d73-9570-45788f871476\") " Oct 30 10:58:31.132727 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:31.132457 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5366129e-7a00-4d73-9570-45788f871476-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5366129e-7a00-4d73-9570-45788f871476-data-science-collector-ta-client-cert\") pod \"5366129e-7a00-4d73-9570-45788f871476\" (UID: \"5366129e-7a00-4d73-9570-45788f871476\") " Oct 30 10:58:31.132727 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:31.132557 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5366129e-7a00-4d73-9570-45788f871476-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5366129e-7a00-4d73-9570-45788f871476-monitoring-tls-certs\") pod \"5366129e-7a00-4d73-9570-45788f871476\" (UID: \"5366129e-7a00-4d73-9570-45788f871476\") " Oct 30 10:58:31.132842 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:31.132815 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5366129e-7a00-4d73-9570-45788f871476-otc-internal" pod "5366129e-7a00-4d73-9570-45788f871476" (UID: "5366129e-7a00-4d73-9570-45788f871476"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 10:58:31.134647 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:31.134612 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5366129e-7a00-4d73-9570-45788f871476-monitoring-tls-certs" pod "5366129e-7a00-4d73-9570-45788f871476" (UID: "5366129e-7a00-4d73-9570-45788f871476"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:58:31.134803 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:31.134684 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5366129e-7a00-4d73-9570-45788f871476-data-science-collector-ta-client-cert" pod "5366129e-7a00-4d73-9570-45788f871476" (UID: "5366129e-7a00-4d73-9570-45788f871476"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:58:31.134947 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:31.134921 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5366129e-7a00-4d73-9570-45788f871476-tls-certs" pod "5366129e-7a00-4d73-9570-45788f871476" (UID: "5366129e-7a00-4d73-9570-45788f871476"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 10:58:31.135055 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:31.134955 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5366129e-7a00-4d73-9570-45788f871476-kube-api-access-scdbs" pod "5366129e-7a00-4d73-9570-45788f871476" (UID: "5366129e-7a00-4d73-9570-45788f871476"). InnerVolumeSpecName "kube-api-access-scdbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 10:58:31.178091 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:31.178060 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:31.233238 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:31.233215 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5366129e-7a00-4d73-9570-45788f871476-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:58:31.233238 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:31.233237 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scdbs\" (UniqueName: \"kubernetes.io/projected/5366129e-7a00-4d73-9570-45788f871476-kube-api-access-scdbs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:58:31.233384 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:31.233248 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5366129e-7a00-4d73-9570-45788f871476-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:58:31.233384 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:31.233259 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5366129e-7a00-4d73-9570-45788f871476-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:58:31.233384 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:31.233268 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5366129e-7a00-4d73-9570-45788f871476-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 10:58:32.024556 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.024509 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5366129e-7a00-4d73-9570-45788f871476","Type":"ContainerDied","Data":"f8be5879da54b637f108686ac22f7916444a16ecbb681b6710c817a59bf04509"} Oct 30 10:58:32.025080 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.024572 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:32.025080 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.024611 4289 scope.go:122] "RemoveContainer" containerID="2815f69c90d4e91012f46ac314e91f86d77ed421d117527d2f666ba590740561" Oct 30 10:58:32.072783 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.072749 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:58:32.084483 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.084461 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:58:32.112936 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.112908 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:58:32.113303 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.113290 4289 cpu_manager.go:414] "containerMap: removing container" podUID="5366129e-7a00-4d73-9570-45788f871476" containerName="otc-container" Oct 30 10:58:32.113345 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.113304 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5366129e-7a00-4d73-9570-45788f871476" containerName="otc-container" Oct 30 10:58:32.113389 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.113380 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="5366129e-7a00-4d73-9570-45788f871476" containerName="otc-container" Oct 30 10:58:32.118132 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.118113 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:32.134455 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.134413 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 10:58:32.134639 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.134511 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 10:58:32.134639 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.134578 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 10:58:32.134924 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.134905 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-e4cadd38\"" Oct 30 10:58:32.134997 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.134984 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8vrjb\"" Oct 30 10:58:32.136976 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.136954 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:58:32.139649 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.139626 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d2lh\" (UniqueName: \"kubernetes.io/projected/2687b5a7-06b4-4e94-9ace-232127c3c7ab-kube-api-access-9d2lh\") pod \"data-science-collector-collector-1\" (UID: \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:32.139714 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.139664 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2687b5a7-06b4-4e94-9ace-232127c3c7ab-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:32.139714 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.139685 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2687b5a7-06b4-4e94-9ace-232127c3c7ab-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:32.139814 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.139797 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2687b5a7-06b4-4e94-9ace-232127c3c7ab-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:32.139850 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.139829 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2687b5a7-06b4-4e94-9ace-232127c3c7ab-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:32.241067 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.241037 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d2lh\" (UniqueName: \"kubernetes.io/projected/2687b5a7-06b4-4e94-9ace-232127c3c7ab-kube-api-access-9d2lh\") pod \"data-science-collector-collector-1\" (UID: \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:32.241239 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.241079 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2687b5a7-06b4-4e94-9ace-232127c3c7ab-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:32.241239 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.241109 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2687b5a7-06b4-4e94-9ace-232127c3c7ab-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:32.241239 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.241156 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2687b5a7-06b4-4e94-9ace-232127c3c7ab-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:32.241239 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.241181 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2687b5a7-06b4-4e94-9ace-232127c3c7ab-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:32.241862 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.241837 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2687b5a7-06b4-4e94-9ace-232127c3c7ab-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:32.243636 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.243613 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2687b5a7-06b4-4e94-9ace-232127c3c7ab-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:32.243732 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.243646 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2687b5a7-06b4-4e94-9ace-232127c3c7ab-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:32.243732 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.243681 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2687b5a7-06b4-4e94-9ace-232127c3c7ab-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:32.263280 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.263246 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d2lh\" (UniqueName: \"kubernetes.io/projected/2687b5a7-06b4-4e94-9ace-232127c3c7ab-kube-api-access-9d2lh\") pod \"data-science-collector-collector-1\" (UID: \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:32.430937 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.430860 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 10:58:32.725373 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.725346 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 10:58:32.725555 ip-10-0-137-242 kubenswrapper[4289]: W1030 10:58:32.725533 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2687b5a7_06b4_4e94_9ace_232127c3c7ab.slice/crio-ce77f9cb0c6995353e06e7bd6d9b252428e4a7d50375c3afd04cb33111125831 WatchSource:0}: Error finding container ce77f9cb0c6995353e06e7bd6d9b252428e4a7d50375c3afd04cb33111125831: Status 404 returned error can't find the container with id ce77f9cb0c6995353e06e7bd6d9b252428e4a7d50375c3afd04cb33111125831 Oct 30 10:58:32.727789 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.727751 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:58:32.727853 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:32.727815 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 10:58:33.032167 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:33.032085 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2687b5a7-06b4-4e94-9ace-232127c3c7ab","Type":"ContainerStarted","Data":"619c45aca7735cb6432336157bd3f9c902272605345f4b0bbb7842208ba0be67"} Oct 30 10:58:33.032167 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:33.032145 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2687b5a7-06b4-4e94-9ace-232127c3c7ab","Type":"ContainerStarted","Data":"ce77f9cb0c6995353e06e7bd6d9b252428e4a7d50375c3afd04cb33111125831"} Oct 30 10:58:33.073635 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:33.073580 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.073566524 podStartE2EDuration="1.073566524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 10:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 10:58:33.072172074 +0000 UTC m=+803.573551559" watchObservedRunningTime="2026-10-30 10:58:33.073566524 +0000 UTC m=+803.574946008" Oct 30 10:58:33.557289 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:33.557229 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:33.624884 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:33.624853 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5366129e-7a00-4d73-9570-45788f871476" path="/var/lib/kubelet/pods/5366129e-7a00-4d73-9570-45788f871476/volumes" Oct 30 10:58:33.740057 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:33.739998 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:50.875494 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:50.875454 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:51.036203 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:51.036164 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:54.926417 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:58:54.926349 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:58:56.257340 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:58:56.257305 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:59:22.487465 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:59:22.487394 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:59:22.650600 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:59:22.650518 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:59:26.257854 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:59:26.257809 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 10:59:40.502859 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:59:40.502825 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:59:40.671423 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:59:40.671384 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:59:46.322166 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:59:46.322121 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:59:46.498874 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:59:46.498810 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:59:46.664687 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:59:46.664611 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:59:52.198147 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:59:52.198102 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:59:52.440564 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:59:52.440528 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:59:52.812293 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:59:52.812256 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:59:52.977979 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:59:52.977935 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:59:55.037065 ip-10-0-137-242 kubenswrapper[4289]: I1030 10:59:55.037028 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 10:59:56.257270 ip-10-0-137-242 kubenswrapper[4289]: E1030 10:59:56.257230 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:00:03.832314 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:03.832273 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:00:04.020264 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:04.020218 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:00:04.186177 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:04.186088 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:00:05.565023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:05.564987 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:00:05.565520 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:05.565245 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="2687b5a7-06b4-4e94-9ace-232127c3c7ab" containerName="otc-container" containerID="cri-o://619c45aca7735cb6432336157bd3f9c902272605345f4b0bbb7842208ba0be67" gracePeriod=30 Oct 30 11:00:05.750103 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:05.750069 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:00:05.919156 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:05.919067 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:00:06.076320 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:06.076284 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:00:07.915397 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:07.915375 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:00:07.975823 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:07.975797 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2687b5a7-06b4-4e94-9ace-232127c3c7ab-otc-internal\" (UniqueName: \"kubernetes.io/configmap/2687b5a7-06b4-4e94-9ace-232127c3c7ab-otc-internal\") pod \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\" (UID: \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\") " Oct 30 11:00:07.975970 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:07.975897 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2687b5a7-06b4-4e94-9ace-232127c3c7ab-kube-api-access-9d2lh\" (UniqueName: \"kubernetes.io/projected/2687b5a7-06b4-4e94-9ace-232127c3c7ab-kube-api-access-9d2lh\") pod \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\" (UID: \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\") " Oct 30 11:00:07.975970 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:07.975918 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2687b5a7-06b4-4e94-9ace-232127c3c7ab-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2687b5a7-06b4-4e94-9ace-232127c3c7ab-data-science-collector-ta-client-cert\") pod \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\" (UID: \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\") " Oct 30 11:00:07.975970 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:07.975946 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2687b5a7-06b4-4e94-9ace-232127c3c7ab-tls-certs\" (UniqueName: \"kubernetes.io/secret/2687b5a7-06b4-4e94-9ace-232127c3c7ab-tls-certs\") pod \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\" (UID: \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\") " Oct 30 11:00:07.976143 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:07.976108 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2687b5a7-06b4-4e94-9ace-232127c3c7ab-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2687b5a7-06b4-4e94-9ace-232127c3c7ab-monitoring-tls-certs\") pod \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\" (UID: \"2687b5a7-06b4-4e94-9ace-232127c3c7ab\") " Oct 30 11:00:07.976303 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:07.976276 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2687b5a7-06b4-4e94-9ace-232127c3c7ab-otc-internal" pod "2687b5a7-06b4-4e94-9ace-232127c3c7ab" (UID: "2687b5a7-06b4-4e94-9ace-232127c3c7ab"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:00:07.976441 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:07.976411 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2687b5a7-06b4-4e94-9ace-232127c3c7ab-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:00:07.978495 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:07.978384 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2687b5a7-06b4-4e94-9ace-232127c3c7ab-data-science-collector-ta-client-cert" pod "2687b5a7-06b4-4e94-9ace-232127c3c7ab" (UID: "2687b5a7-06b4-4e94-9ace-232127c3c7ab"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:00:07.978495 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:07.978416 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2687b5a7-06b4-4e94-9ace-232127c3c7ab-monitoring-tls-certs" pod "2687b5a7-06b4-4e94-9ace-232127c3c7ab" (UID: "2687b5a7-06b4-4e94-9ace-232127c3c7ab"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:00:07.978632 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:07.978510 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2687b5a7-06b4-4e94-9ace-232127c3c7ab-tls-certs" pod "2687b5a7-06b4-4e94-9ace-232127c3c7ab" (UID: "2687b5a7-06b4-4e94-9ace-232127c3c7ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:00:07.978675 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:07.978622 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2687b5a7-06b4-4e94-9ace-232127c3c7ab-kube-api-access-9d2lh" pod "2687b5a7-06b4-4e94-9ace-232127c3c7ab" (UID: "2687b5a7-06b4-4e94-9ace-232127c3c7ab"). InnerVolumeSpecName "kube-api-access-9d2lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:00:08.077583 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:08.077542 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d2lh\" (UniqueName: \"kubernetes.io/projected/2687b5a7-06b4-4e94-9ace-232127c3c7ab-kube-api-access-9d2lh\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:00:08.077583 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:08.077583 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2687b5a7-06b4-4e94-9ace-232127c3c7ab-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:00:08.077787 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:08.077601 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2687b5a7-06b4-4e94-9ace-232127c3c7ab-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:00:08.077787 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:08.077617 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2687b5a7-06b4-4e94-9ace-232127c3c7ab-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:00:08.469048 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:08.468963 4289 generic.go:356] "Generic (PLEG): container finished" podID="2687b5a7-06b4-4e94-9ace-232127c3c7ab" containerID="619c45aca7735cb6432336157bd3f9c902272605345f4b0bbb7842208ba0be67" exitCode=0 Oct 30 11:00:08.469048 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:08.469011 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:00:08.469240 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:08.469046 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2687b5a7-06b4-4e94-9ace-232127c3c7ab","Type":"ContainerDied","Data":"619c45aca7735cb6432336157bd3f9c902272605345f4b0bbb7842208ba0be67"} Oct 30 11:00:08.469240 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:08.469100 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2687b5a7-06b4-4e94-9ace-232127c3c7ab","Type":"ContainerDied","Data":"ce77f9cb0c6995353e06e7bd6d9b252428e4a7d50375c3afd04cb33111125831"} Oct 30 11:00:08.469240 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:08.469124 4289 scope.go:122] "RemoveContainer" containerID="619c45aca7735cb6432336157bd3f9c902272605345f4b0bbb7842208ba0be67" Oct 30 11:00:08.478219 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:08.478201 4289 scope.go:122] "RemoveContainer" containerID="619c45aca7735cb6432336157bd3f9c902272605345f4b0bbb7842208ba0be67" Oct 30 11:00:08.478491 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:00:08.478473 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"619c45aca7735cb6432336157bd3f9c902272605345f4b0bbb7842208ba0be67\": container with ID starting with 619c45aca7735cb6432336157bd3f9c902272605345f4b0bbb7842208ba0be67 not found: ID does not exist" containerID="619c45aca7735cb6432336157bd3f9c902272605345f4b0bbb7842208ba0be67" Oct 30 11:00:08.478592 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:08.478498 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"619c45aca7735cb6432336157bd3f9c902272605345f4b0bbb7842208ba0be67"} err="failed to get container status \"619c45aca7735cb6432336157bd3f9c902272605345f4b0bbb7842208ba0be67\": rpc error: code = NotFound desc = could not find container \"619c45aca7735cb6432336157bd3f9c902272605345f4b0bbb7842208ba0be67\": container with ID starting with 619c45aca7735cb6432336157bd3f9c902272605345f4b0bbb7842208ba0be67 not found: ID does not exist" Oct 30 11:00:08.520192 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:08.520165 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:00:08.532059 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:08.532039 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:00:09.627036 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:09.626994 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2687b5a7-06b4-4e94-9ace-232127c3c7ab" path="/var/lib/kubelet/pods/2687b5a7-06b4-4e94-9ace-232127c3c7ab/volumes" Oct 30 11:00:26.257650 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:00:26.257614 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:00:28.037240 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:28.037198 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:00:55.106938 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:00:55.106851 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:00:56.257947 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:00:56.257913 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:01:22.705076 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:01:22.705042 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:01:26.257128 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:01:26.257091 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:01:40.305352 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:01:40.305317 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:01:40.507977 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:01:40.507928 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:01:40.677766 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:01:40.677654 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:01:42.282483 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:01:42.282448 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:01:42.448584 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:01:42.448534 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:01:44.177062 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:01:44.177014 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:01:49.213065 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:01:49.213028 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:01:54.369889 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:01:54.369849 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:01:55.157146 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:01:55.157104 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:01:55.336920 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:01:55.336882 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:01:56.257174 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:01:56.257139 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:01:59.507078 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:01:59.507041 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:01:59.707173 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:01:59.707127 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:01:59.902073 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:01:59.901980 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:02:01.419416 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:02:01.419381 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:02:01.604959 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:02:01.604924 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:02:02.725351 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:02:02.725317 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:02:02.910409 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:02:02.910363 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:02:03.077979 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:02:03.077876 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:02:26.257883 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:02:26.257800 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:02:40.424762 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:02:40.424728 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:02:40.600189 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:02:40.600155 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:02:40.766052 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:02:40.766018 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:02:55.395106 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:02:55.395064 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:02:56.259478 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:02:56.259419 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:03:26.258243 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:03:26.258208 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:03:55.512887 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:03:55.512800 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:03:56.257561 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:03:56.257527 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:04:26.257130 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:04:26.257090 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:04:55.637377 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:04:55.637339 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:04:56.257253 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:04:56.257214 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:05:26.256985 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:05:26.256897 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:05:39.904244 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:05:39.904205 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:05:45.460960 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:05:45.460922 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:05:45.623140 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:05:45.623067 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:05:46.003179 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:05:46.003133 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:05:46.165587 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:05:46.165558 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:05:51.752913 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:05:51.752878 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:05:51.920980 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:05:51.920940 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:05:52.132677 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:05:52.132584 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:05:55.757664 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:05:55.757623 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:05:56.258220 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:05:56.258185 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:05:57.748560 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:05:57.748514 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:05:57.973558 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:05:57.973522 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:06:00.713541 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:00.713509 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:06:00.914091 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:00.914055 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:06:01.115009 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:01.114962 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:06:02.587712 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:02.587672 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:06:02.751355 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:02.751301 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:06:04.363410 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:04.363372 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:06:05.836143 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:05.836108 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:06:05.836547 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:05.836533 4289 cpu_manager.go:414] "containerMap: removing container" podUID="2687b5a7-06b4-4e94-9ace-232127c3c7ab" containerName="otc-container" Oct 30 11:06:05.836592 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:05.836547 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2687b5a7-06b4-4e94-9ace-232127c3c7ab" containerName="otc-container" Oct 30 11:06:05.836626 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:05.836613 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="2687b5a7-06b4-4e94-9ace-232127c3c7ab" containerName="otc-container" Oct 30 11:06:05.839670 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:05.839653 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:06:05.868246 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:05.868214 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 11:06:05.868447 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:05.868334 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 11:06:05.868447 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:05.868392 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 11:06:05.868573 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:05.868484 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8vrjb\"" Oct 30 11:06:05.880445 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:05.880409 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-e4cadd38\"" Oct 30 11:06:05.883086 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:05.883062 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:06:05.972064 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:05.972030 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr5xm\" (UniqueName: \"kubernetes.io/projected/845b7403-ad65-448e-97d4-6fb29627e099-kube-api-access-kr5xm\") pod \"data-science-collector-collector-1\" (UID: \"845b7403-ad65-448e-97d4-6fb29627e099\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:06:05.972239 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:05.972081 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/845b7403-ad65-448e-97d4-6fb29627e099-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"845b7403-ad65-448e-97d4-6fb29627e099\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:06:05.972239 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:05.972183 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/845b7403-ad65-448e-97d4-6fb29627e099-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"845b7403-ad65-448e-97d4-6fb29627e099\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:06:05.972320 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:05.972260 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/845b7403-ad65-448e-97d4-6fb29627e099-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"845b7403-ad65-448e-97d4-6fb29627e099\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:06:05.972320 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:05.972282 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/845b7403-ad65-448e-97d4-6fb29627e099-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"845b7403-ad65-448e-97d4-6fb29627e099\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:06:06.072817 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:06.072786 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr5xm\" (UniqueName: \"kubernetes.io/projected/845b7403-ad65-448e-97d4-6fb29627e099-kube-api-access-kr5xm\") pod \"data-science-collector-collector-1\" (UID: \"845b7403-ad65-448e-97d4-6fb29627e099\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:06:06.073013 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:06.072828 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/845b7403-ad65-448e-97d4-6fb29627e099-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"845b7403-ad65-448e-97d4-6fb29627e099\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:06:06.073013 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:06.072848 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/845b7403-ad65-448e-97d4-6fb29627e099-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"845b7403-ad65-448e-97d4-6fb29627e099\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:06:06.073013 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:06.072891 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/845b7403-ad65-448e-97d4-6fb29627e099-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"845b7403-ad65-448e-97d4-6fb29627e099\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:06:06.073013 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:06.072906 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/845b7403-ad65-448e-97d4-6fb29627e099-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"845b7403-ad65-448e-97d4-6fb29627e099\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:06:06.073718 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:06.073693 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/845b7403-ad65-448e-97d4-6fb29627e099-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"845b7403-ad65-448e-97d4-6fb29627e099\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:06:06.075331 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:06.075308 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/845b7403-ad65-448e-97d4-6fb29627e099-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"845b7403-ad65-448e-97d4-6fb29627e099\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:06:06.075447 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:06.075398 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/845b7403-ad65-448e-97d4-6fb29627e099-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"845b7403-ad65-448e-97d4-6fb29627e099\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:06:06.075545 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:06.075527 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/845b7403-ad65-448e-97d4-6fb29627e099-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"845b7403-ad65-448e-97d4-6fb29627e099\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:06:06.095841 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:06.095768 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr5xm\" (UniqueName: \"kubernetes.io/projected/845b7403-ad65-448e-97d4-6fb29627e099-kube-api-access-kr5xm\") pod \"data-science-collector-collector-1\" (UID: \"845b7403-ad65-448e-97d4-6fb29627e099\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:06:06.153154 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:06.153126 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:06:06.445714 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:06.445681 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:06:06.446008 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:06:06.445986 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845b7403_ad65_448e_97d4_6fb29627e099.slice/crio-f3101be966cb65d825c9a76e076b2e19a63977cd8706c9e03f76468d8d0cfcde WatchSource:0}: Error finding container f3101be966cb65d825c9a76e076b2e19a63977cd8706c9e03f76468d8d0cfcde: Status 404 returned error can't find the container with id f3101be966cb65d825c9a76e076b2e19a63977cd8706c9e03f76468d8d0cfcde Oct 30 11:06:06.448213 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:06.448183 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:06:06.448282 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:06.448248 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:06:07.035411 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:07.035376 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"845b7403-ad65-448e-97d4-6fb29627e099","Type":"ContainerStarted","Data":"b8abbf1f8aca8eab9fb7ea6f4cbfab39e2212f497bb91472bb803b41addbaa76"} Oct 30 11:06:07.035916 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:07.035449 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"845b7403-ad65-448e-97d4-6fb29627e099","Type":"ContainerStarted","Data":"f3101be966cb65d825c9a76e076b2e19a63977cd8706c9e03f76468d8d0cfcde"} Oct 30 11:06:07.076303 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:07.076248 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.076229842 podStartE2EDuration="2.076229842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:06:07.075707299 +0000 UTC m=+1257.577086797" watchObservedRunningTime="2026-10-30 11:06:07.076229842 +0000 UTC m=+1257.577609326" Oct 30 11:06:26.258574 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:06:26.258527 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:06:38.170863 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:38.170816 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:06:38.331228 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:38.331196 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:06:56.257905 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:06:56.257703 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:06:57.577844 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:06:57.577803 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:07:04.431661 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:07:04.431619 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:07:26.257564 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:07:26.257525 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:07:34.174976 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:07:34.174945 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:07:42.109802 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:07:42.109759 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:07:42.270723 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:07:42.270665 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:07:49.873091 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:07:49.873001 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:07:56.000265 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:07:56.000218 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:07:56.257921 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:07:56.257832 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:08:26.257939 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:08:26.257859 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:08:56.136389 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:08:56.136351 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:08:56.257899 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:08:56.257864 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:09:26.257712 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:09:26.257669 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:09:56.247622 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:09:56.247580 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:09:56.258775 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:09:56.258745 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:10:26.257703 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:10:26.257669 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:10:56.258162 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:10:56.258126 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:10:56.445191 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:10:56.445151 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:10:58.136606 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:10:58.136568 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:10:58.371218 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:10:58.371176 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:04.100454 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:04.100398 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:04.291969 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:04.291935 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:13.492322 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:13.492278 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:13.677504 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:13.677474 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:16.046465 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:16.046408 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:26.257441 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:11:26.257347 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:11:30.397112 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:30.397077 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:33.589575 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:33.589544 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8"] Oct 30 11:11:33.590033 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:33.589788 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" podUID="742466b8-3c11-4a49-90bd-08aae667cdaf" containerName="kube-rbac-proxy" containerID="cri-o://ae6a182049bc84539c34da26bceed11b915379a0ca77c23f8bd562000d04d8aa" gracePeriod=30 Oct 30 11:11:33.601451 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:33.601402 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks"] Oct 30 11:11:33.601718 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:33.601693 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" podUID="073dc996-75bd-41d8-b531-93aaf2ba4b12" containerName="kube-rbac-proxy" containerID="cri-o://bb4657afdd237952f25127416e3df9bd958c4868c437255a98c7b4b4b575a96f" gracePeriod=30 Oct 30 11:11:33.601804 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:33.601697 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" podUID="073dc996-75bd-41d8-b531-93aaf2ba4b12" containerName="method-gate" containerID="cri-o://142073b844cc022f6e0c6f06e3a3818e9720584cb5b8762715e09fc3503f2c95" gracePeriod=30 Oct 30 11:11:33.601865 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:33.601697 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" podUID="073dc996-75bd-41d8-b531-93aaf2ba4b12" containerName="prom-label-proxy" containerID="cri-o://9c1f27e1d764132f027948096dc8902f2b3cf898f1e6a18072627826fb505085" gracePeriod=30 Oct 30 11:11:33.628342 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:33.628312 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9"] Oct 30 11:11:33.628577 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:33.628546 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" podUID="b4822533-f28b-4e97-959b-4e377442ddfd" containerName="kube-rbac-proxy" containerID="cri-o://51fbc7c5dc57675c0ef2dd867a9011b13758074dc11f33c974b987186e2128d8" gracePeriod=30 Oct 30 11:11:33.628577 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:33.628562 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" podUID="b4822533-f28b-4e97-959b-4e377442ddfd" containerName="method-gate" containerID="cri-o://6a3c9d6f35ac1b0ed4912648c65edd9bac75bbcf3336d76ea90ac184f45eb826" gracePeriod=30 Oct 30 11:11:33.628772 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:33.628562 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" podUID="b4822533-f28b-4e97-959b-4e377442ddfd" containerName="prom-label-proxy" containerID="cri-o://b2386b3362d81b4da34ca01a5d45d94caa39807da09c80e94564bf55498256f0" gracePeriod=30 Oct 30 11:11:33.643617 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:33.643591 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8"] Oct 30 11:11:33.643870 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:33.643835 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8" podUID="4a5f36fe-3925-420e-9a5f-1e2d1e70a78e" containerName="korrel8r" containerID="cri-o://266399455f3ea9c4b4a38815d8756bc7091879d59d7971ba2cb2661f48f83ae5" gracePeriod=30 Oct 30 11:11:33.891707 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:33.891679 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" Oct 30 11:11:33.923374 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:33.922576 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4822533-f28b-4e97-959b-4e377442ddfd-config\" (UniqueName: \"kubernetes.io/configmap/b4822533-f28b-4e97-959b-4e377442ddfd-config\") pod \"b4822533-f28b-4e97-959b-4e377442ddfd\" (UID: \"b4822533-f28b-4e97-959b-4e377442ddfd\") " Oct 30 11:11:33.923374 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:33.922648 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4822533-f28b-4e97-959b-4e377442ddfd-tls\" (UniqueName: \"kubernetes.io/secret/b4822533-f28b-4e97-959b-4e377442ddfd-tls\") pod \"b4822533-f28b-4e97-959b-4e377442ddfd\" (UID: \"b4822533-f28b-4e97-959b-4e377442ddfd\") " Oct 30 11:11:33.923374 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:33.922738 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4822533-f28b-4e97-959b-4e377442ddfd-kube-api-access-kzdzj\" (UniqueName: \"kubernetes.io/projected/b4822533-f28b-4e97-959b-4e377442ddfd-kube-api-access-kzdzj\") pod \"b4822533-f28b-4e97-959b-4e377442ddfd\" (UID: \"b4822533-f28b-4e97-959b-4e377442ddfd\") " Oct 30 11:11:33.923770 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:33.923727 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4822533-f28b-4e97-959b-4e377442ddfd-config" pod "b4822533-f28b-4e97-959b-4e377442ddfd" (UID: "b4822533-f28b-4e97-959b-4e377442ddfd"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:11:33.927411 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:33.926799 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4822533-f28b-4e97-959b-4e377442ddfd-kube-api-access-kzdzj" pod "b4822533-f28b-4e97-959b-4e377442ddfd" (UID: "b4822533-f28b-4e97-959b-4e377442ddfd"). InnerVolumeSpecName "kube-api-access-kzdzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:11:33.927664 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:33.927578 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4822533-f28b-4e97-959b-4e377442ddfd-tls" pod "b4822533-f28b-4e97-959b-4e377442ddfd" (UID: "b4822533-f28b-4e97-959b-4e377442ddfd"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:11:33.952031 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:33.947999 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" Oct 30 11:11:33.979687 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:33.979661 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" Oct 30 11:11:33.986378 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:33.986351 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8" Oct 30 11:11:34.024259 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.024226 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/742466b8-3c11-4a49-90bd-08aae667cdaf-kube-api-access-mr8st\" (UniqueName: \"kubernetes.io/projected/742466b8-3c11-4a49-90bd-08aae667cdaf-kube-api-access-mr8st\") pod \"742466b8-3c11-4a49-90bd-08aae667cdaf\" (UID: \"742466b8-3c11-4a49-90bd-08aae667cdaf\") " Oct 30 11:11:34.024452 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.024305 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a5f36fe-3925-420e-9a5f-1e2d1e70a78e-tls\" (UniqueName: \"kubernetes.io/secret/4a5f36fe-3925-420e-9a5f-1e2d1e70a78e-tls\") pod \"4a5f36fe-3925-420e-9a5f-1e2d1e70a78e\" (UID: \"4a5f36fe-3925-420e-9a5f-1e2d1e70a78e\") " Oct 30 11:11:34.024452 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.024347 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a5f36fe-3925-420e-9a5f-1e2d1e70a78e-config\" (UniqueName: \"kubernetes.io/configmap/4a5f36fe-3925-420e-9a5f-1e2d1e70a78e-config\") pod \"4a5f36fe-3925-420e-9a5f-1e2d1e70a78e\" (UID: \"4a5f36fe-3925-420e-9a5f-1e2d1e70a78e\") " Oct 30 11:11:34.024452 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.024377 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/742466b8-3c11-4a49-90bd-08aae667cdaf-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/742466b8-3c11-4a49-90bd-08aae667cdaf-prometheus-ca\") pod \"742466b8-3c11-4a49-90bd-08aae667cdaf\" (UID: \"742466b8-3c11-4a49-90bd-08aae667cdaf\") " Oct 30 11:11:34.024636 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.024505 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/073dc996-75bd-41d8-b531-93aaf2ba4b12-config\" (UniqueName: \"kubernetes.io/configmap/073dc996-75bd-41d8-b531-93aaf2ba4b12-config\") pod \"073dc996-75bd-41d8-b531-93aaf2ba4b12\" (UID: \"073dc996-75bd-41d8-b531-93aaf2ba4b12\") " Oct 30 11:11:34.024636 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.024557 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/073dc996-75bd-41d8-b531-93aaf2ba4b12-client-ca\" (UniqueName: \"kubernetes.io/configmap/073dc996-75bd-41d8-b531-93aaf2ba4b12-client-ca\") pod \"073dc996-75bd-41d8-b531-93aaf2ba4b12\" (UID: \"073dc996-75bd-41d8-b531-93aaf2ba4b12\") " Oct 30 11:11:34.024636 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.024602 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/742466b8-3c11-4a49-90bd-08aae667cdaf-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/742466b8-3c11-4a49-90bd-08aae667cdaf-kube-rbac-proxy-config\") pod \"742466b8-3c11-4a49-90bd-08aae667cdaf\" (UID: \"742466b8-3c11-4a49-90bd-08aae667cdaf\") " Oct 30 11:11:34.024792 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.024647 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/073dc996-75bd-41d8-b531-93aaf2ba4b12-kube-api-access-5sdzk\" (UniqueName: \"kubernetes.io/projected/073dc996-75bd-41d8-b531-93aaf2ba4b12-kube-api-access-5sdzk\") pod \"073dc996-75bd-41d8-b531-93aaf2ba4b12\" (UID: \"073dc996-75bd-41d8-b531-93aaf2ba4b12\") " Oct 30 11:11:34.024792 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.024676 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/742466b8-3c11-4a49-90bd-08aae667cdaf-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/742466b8-3c11-4a49-90bd-08aae667cdaf-prometheus-client-cert\") pod \"742466b8-3c11-4a49-90bd-08aae667cdaf\" (UID: \"742466b8-3c11-4a49-90bd-08aae667cdaf\") " Oct 30 11:11:34.024792 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.024733 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a5f36fe-3925-420e-9a5f-1e2d1e70a78e-kube-api-access-mhrc4\" (UniqueName: \"kubernetes.io/projected/4a5f36fe-3925-420e-9a5f-1e2d1e70a78e-kube-api-access-mhrc4\") pod \"4a5f36fe-3925-420e-9a5f-1e2d1e70a78e\" (UID: \"4a5f36fe-3925-420e-9a5f-1e2d1e70a78e\") " Oct 30 11:11:34.024792 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.024753 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/742466b8-3c11-4a49-90bd-08aae667cdaf-prometheus-ca" pod "742466b8-3c11-4a49-90bd-08aae667cdaf" (UID: "742466b8-3c11-4a49-90bd-08aae667cdaf"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:11:34.024792 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.024767 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/742466b8-3c11-4a49-90bd-08aae667cdaf-tls\" (UniqueName: \"kubernetes.io/secret/742466b8-3c11-4a49-90bd-08aae667cdaf-tls\") pod \"742466b8-3c11-4a49-90bd-08aae667cdaf\" (UID: \"742466b8-3c11-4a49-90bd-08aae667cdaf\") " Oct 30 11:11:34.025063 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.024810 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/073dc996-75bd-41d8-b531-93aaf2ba4b12-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/073dc996-75bd-41d8-b531-93aaf2ba4b12-prometheus-tls-ca\") pod \"073dc996-75bd-41d8-b531-93aaf2ba4b12\" (UID: \"073dc996-75bd-41d8-b531-93aaf2ba4b12\") " Oct 30 11:11:34.025063 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.024848 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/073dc996-75bd-41d8-b531-93aaf2ba4b12-tls\" (UniqueName: \"kubernetes.io/secret/073dc996-75bd-41d8-b531-93aaf2ba4b12-tls\") pod \"073dc996-75bd-41d8-b531-93aaf2ba4b12\" (UID: \"073dc996-75bd-41d8-b531-93aaf2ba4b12\") " Oct 30 11:11:34.025063 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.024988 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/073dc996-75bd-41d8-b531-93aaf2ba4b12-client-ca" pod "073dc996-75bd-41d8-b531-93aaf2ba4b12" (UID: "073dc996-75bd-41d8-b531-93aaf2ba4b12"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:11:34.025223 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.025170 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzdzj\" (UniqueName: \"kubernetes.io/projected/b4822533-f28b-4e97-959b-4e377442ddfd-kube-api-access-kzdzj\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:34.025223 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.025192 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4822533-f28b-4e97-959b-4e377442ddfd-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:34.025223 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.025207 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/742466b8-3c11-4a49-90bd-08aae667cdaf-prometheus-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:34.025223 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.025220 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b4822533-f28b-4e97-959b-4e377442ddfd-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:34.025440 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.025232 4289 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/073dc996-75bd-41d8-b531-93aaf2ba4b12-client-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:34.026313 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.025907 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/073dc996-75bd-41d8-b531-93aaf2ba4b12-prometheus-tls-ca" pod "073dc996-75bd-41d8-b531-93aaf2ba4b12" (UID: "073dc996-75bd-41d8-b531-93aaf2ba4b12"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:11:34.026313 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.025918 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/073dc996-75bd-41d8-b531-93aaf2ba4b12-config" pod "073dc996-75bd-41d8-b531-93aaf2ba4b12" (UID: "073dc996-75bd-41d8-b531-93aaf2ba4b12"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:11:34.026558 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.026356 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a5f36fe-3925-420e-9a5f-1e2d1e70a78e-config" pod "4a5f36fe-3925-420e-9a5f-1e2d1e70a78e" (UID: "4a5f36fe-3925-420e-9a5f-1e2d1e70a78e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:11:34.028452 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.028358 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a5f36fe-3925-420e-9a5f-1e2d1e70a78e-tls" pod "4a5f36fe-3925-420e-9a5f-1e2d1e70a78e" (UID: "4a5f36fe-3925-420e-9a5f-1e2d1e70a78e"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:11:34.028810 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.028759 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/742466b8-3c11-4a49-90bd-08aae667cdaf-tls" pod "742466b8-3c11-4a49-90bd-08aae667cdaf" (UID: "742466b8-3c11-4a49-90bd-08aae667cdaf"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:11:34.029091 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.029058 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073dc996-75bd-41d8-b531-93aaf2ba4b12-kube-api-access-5sdzk" pod "073dc996-75bd-41d8-b531-93aaf2ba4b12" (UID: "073dc996-75bd-41d8-b531-93aaf2ba4b12"). InnerVolumeSpecName "kube-api-access-5sdzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:11:34.029091 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.029075 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/073dc996-75bd-41d8-b531-93aaf2ba4b12-tls" pod "073dc996-75bd-41d8-b531-93aaf2ba4b12" (UID: "073dc996-75bd-41d8-b531-93aaf2ba4b12"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:11:34.029270 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.029241 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/742466b8-3c11-4a49-90bd-08aae667cdaf-kube-rbac-proxy-config" pod "742466b8-3c11-4a49-90bd-08aae667cdaf" (UID: "742466b8-3c11-4a49-90bd-08aae667cdaf"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:11:34.029717 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.029694 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/742466b8-3c11-4a49-90bd-08aae667cdaf-kube-api-access-mr8st" pod "742466b8-3c11-4a49-90bd-08aae667cdaf" (UID: "742466b8-3c11-4a49-90bd-08aae667cdaf"). InnerVolumeSpecName "kube-api-access-mr8st". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:11:34.030988 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.030967 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/742466b8-3c11-4a49-90bd-08aae667cdaf-prometheus-client-cert" pod "742466b8-3c11-4a49-90bd-08aae667cdaf" (UID: "742466b8-3c11-4a49-90bd-08aae667cdaf"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:11:34.031075 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.031003 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a5f36fe-3925-420e-9a5f-1e2d1e70a78e-kube-api-access-mhrc4" pod "4a5f36fe-3925-420e-9a5f-1e2d1e70a78e" (UID: "4a5f36fe-3925-420e-9a5f-1e2d1e70a78e"). InnerVolumeSpecName "kube-api-access-mhrc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:11:34.056726 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.052274 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:34.126037 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.125954 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/742466b8-3c11-4a49-90bd-08aae667cdaf-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:34.126037 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.125993 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/073dc996-75bd-41d8-b531-93aaf2ba4b12-prometheus-tls-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:34.126037 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.126008 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/073dc996-75bd-41d8-b531-93aaf2ba4b12-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:34.126037 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.126025 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr8st\" (UniqueName: \"kubernetes.io/projected/742466b8-3c11-4a49-90bd-08aae667cdaf-kube-api-access-mr8st\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:34.126037 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.126039 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4a5f36fe-3925-420e-9a5f-1e2d1e70a78e-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:34.126323 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.126052 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a5f36fe-3925-420e-9a5f-1e2d1e70a78e-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:34.126323 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.126065 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/073dc996-75bd-41d8-b531-93aaf2ba4b12-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:34.126323 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.126078 4289 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/742466b8-3c11-4a49-90bd-08aae667cdaf-kube-rbac-proxy-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:34.126323 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.126091 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sdzk\" (UniqueName: \"kubernetes.io/projected/073dc996-75bd-41d8-b531-93aaf2ba4b12-kube-api-access-5sdzk\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:34.126323 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.126106 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/742466b8-3c11-4a49-90bd-08aae667cdaf-prometheus-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:34.126323 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.126120 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhrc4\" (UniqueName: \"kubernetes.io/projected/4a5f36fe-3925-420e-9a5f-1e2d1e70a78e-kube-api-access-mhrc4\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:34.128203 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.128175 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:11:34.128582 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.128554 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="8ab3b83f-baa8-447b-ad20-76c66c15a7f8" containerName="prometheus" containerID="cri-o://13d58b742224e5fae291138650bc03a4f617f437fb6e647d9f7961d144fcbe13" gracePeriod=600 Oct 30 11:11:34.128713 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.128576 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="8ab3b83f-baa8-447b-ad20-76c66c15a7f8" containerName="thanos-sidecar" containerID="cri-o://895d89ae220db3342f65c5ca22e70050b02d4140d094fccde9ed4235a7383c32" gracePeriod=600 Oct 30 11:11:34.128713 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.128576 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="8ab3b83f-baa8-447b-ad20-76c66c15a7f8" containerName="config-reloader" containerID="cri-o://a4ef55cd9710a012596efb4e6c0c877bc8a32b41f27b98eb4393aa79e9593098" gracePeriod=600 Oct 30 11:11:34.142304 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.141658 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:11:34.142304 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.141994 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="75a3f2af-e799-4bfc-bc43-f40ab46c0913" containerName="alertmanager" containerID="cri-o://e02a4ab64b7d12c2be335320aed7c7677c3f3914d674e7f7a3790301c6ba5e61" gracePeriod=120 Oct 30 11:11:34.142304 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.142191 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="75a3f2af-e799-4bfc-bc43-f40ab46c0913" containerName="config-reloader" containerID="cri-o://f4b8bfdabfb4b4303b7e6e83d20b5006d074b9ea0cc0caa97b8bde7d898677d2" gracePeriod=120 Oct 30 11:11:34.242074 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.241748 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:34.432578 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.432549 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:34.501471 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.501441 4289 generic.go:356] "Generic (PLEG): container finished" podID="4a5f36fe-3925-420e-9a5f-1e2d1e70a78e" containerID="266399455f3ea9c4b4a38815d8756bc7091879d59d7971ba2cb2661f48f83ae5" exitCode=0 Oct 30 11:11:34.501651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.501482 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8" Oct 30 11:11:34.501651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.501515 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8" event={"ID":"4a5f36fe-3925-420e-9a5f-1e2d1e70a78e","Type":"ContainerDied","Data":"266399455f3ea9c4b4a38815d8756bc7091879d59d7971ba2cb2661f48f83ae5"} Oct 30 11:11:34.501651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.501573 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8" event={"ID":"4a5f36fe-3925-420e-9a5f-1e2d1e70a78e","Type":"ContainerDied","Data":"705669658f2fb318c485aced9e38f7fbd3bd104271335dfbeaa0c04b467f52a8"} Oct 30 11:11:34.501651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.501602 4289 scope.go:122] "RemoveContainer" containerID="266399455f3ea9c4b4a38815d8756bc7091879d59d7971ba2cb2661f48f83ae5" Oct 30 11:11:34.503948 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.503926 4289 generic.go:356] "Generic (PLEG): container finished" podID="75a3f2af-e799-4bfc-bc43-f40ab46c0913" containerID="f4b8bfdabfb4b4303b7e6e83d20b5006d074b9ea0cc0caa97b8bde7d898677d2" exitCode=0 Oct 30 11:11:34.504078 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.503999 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"75a3f2af-e799-4bfc-bc43-f40ab46c0913","Type":"ContainerDied","Data":"f4b8bfdabfb4b4303b7e6e83d20b5006d074b9ea0cc0caa97b8bde7d898677d2"} Oct 30 11:11:34.505249 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.505225 4289 generic.go:356] "Generic (PLEG): container finished" podID="742466b8-3c11-4a49-90bd-08aae667cdaf" containerID="ae6a182049bc84539c34da26bceed11b915379a0ca77c23f8bd562000d04d8aa" exitCode=0 Oct 30 11:11:34.505348 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.505287 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" Oct 30 11:11:34.505348 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.505293 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" event={"ID":"742466b8-3c11-4a49-90bd-08aae667cdaf","Type":"ContainerDied","Data":"ae6a182049bc84539c34da26bceed11b915379a0ca77c23f8bd562000d04d8aa"} Oct 30 11:11:34.505348 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.505327 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8" event={"ID":"742466b8-3c11-4a49-90bd-08aae667cdaf","Type":"ContainerDied","Data":"8d6a0db23ca893f4a9f94802ad86443b8d7a3967e88cbe447cb07d974dd5fcd1"} Oct 30 11:11:34.509340 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.509323 4289 generic.go:356] "Generic (PLEG): container finished" podID="8ab3b83f-baa8-447b-ad20-76c66c15a7f8" containerID="895d89ae220db3342f65c5ca22e70050b02d4140d094fccde9ed4235a7383c32" exitCode=0 Oct 30 11:11:34.509340 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.509339 4289 generic.go:356] "Generic (PLEG): container finished" podID="8ab3b83f-baa8-447b-ad20-76c66c15a7f8" containerID="a4ef55cd9710a012596efb4e6c0c877bc8a32b41f27b98eb4393aa79e9593098" exitCode=0 Oct 30 11:11:34.509520 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.509348 4289 generic.go:356] "Generic (PLEG): container finished" podID="8ab3b83f-baa8-447b-ad20-76c66c15a7f8" containerID="13d58b742224e5fae291138650bc03a4f617f437fb6e647d9f7961d144fcbe13" exitCode=0 Oct 30 11:11:34.509520 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.509389 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"8ab3b83f-baa8-447b-ad20-76c66c15a7f8","Type":"ContainerDied","Data":"895d89ae220db3342f65c5ca22e70050b02d4140d094fccde9ed4235a7383c32"} Oct 30 11:11:34.509520 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.509506 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"8ab3b83f-baa8-447b-ad20-76c66c15a7f8","Type":"ContainerDied","Data":"a4ef55cd9710a012596efb4e6c0c877bc8a32b41f27b98eb4393aa79e9593098"} Oct 30 11:11:34.509663 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.509548 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"8ab3b83f-baa8-447b-ad20-76c66c15a7f8","Type":"ContainerDied","Data":"13d58b742224e5fae291138650bc03a4f617f437fb6e647d9f7961d144fcbe13"} Oct 30 11:11:34.511026 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.510993 4289 scope.go:122] "RemoveContainer" containerID="266399455f3ea9c4b4a38815d8756bc7091879d59d7971ba2cb2661f48f83ae5" Oct 30 11:11:34.511279 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:11:34.511256 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"266399455f3ea9c4b4a38815d8756bc7091879d59d7971ba2cb2661f48f83ae5\": container with ID starting with 266399455f3ea9c4b4a38815d8756bc7091879d59d7971ba2cb2661f48f83ae5 not found: ID does not exist" containerID="266399455f3ea9c4b4a38815d8756bc7091879d59d7971ba2cb2661f48f83ae5" Oct 30 11:11:34.511341 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.511291 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"266399455f3ea9c4b4a38815d8756bc7091879d59d7971ba2cb2661f48f83ae5"} err="failed to get container status \"266399455f3ea9c4b4a38815d8756bc7091879d59d7971ba2cb2661f48f83ae5\": rpc error: code = NotFound desc = could not find container \"266399455f3ea9c4b4a38815d8756bc7091879d59d7971ba2cb2661f48f83ae5\": container with ID starting with 266399455f3ea9c4b4a38815d8756bc7091879d59d7971ba2cb2661f48f83ae5 not found: ID does not exist" Oct 30 11:11:34.511341 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.511312 4289 scope.go:122] "RemoveContainer" containerID="ae6a182049bc84539c34da26bceed11b915379a0ca77c23f8bd562000d04d8aa" Oct 30 11:11:34.511921 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.511827 4289 generic.go:356] "Generic (PLEG): container finished" podID="073dc996-75bd-41d8-b531-93aaf2ba4b12" containerID="9c1f27e1d764132f027948096dc8902f2b3cf898f1e6a18072627826fb505085" exitCode=0 Oct 30 11:11:34.511921 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.511853 4289 generic.go:356] "Generic (PLEG): container finished" podID="073dc996-75bd-41d8-b531-93aaf2ba4b12" containerID="142073b844cc022f6e0c6f06e3a3818e9720584cb5b8762715e09fc3503f2c95" exitCode=0 Oct 30 11:11:34.511921 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.511865 4289 generic.go:356] "Generic (PLEG): container finished" podID="073dc996-75bd-41d8-b531-93aaf2ba4b12" containerID="bb4657afdd237952f25127416e3df9bd958c4868c437255a98c7b4b4b575a96f" exitCode=0 Oct 30 11:11:34.511921 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.511899 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" Oct 30 11:11:34.512117 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.511916 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" event={"ID":"073dc996-75bd-41d8-b531-93aaf2ba4b12","Type":"ContainerDied","Data":"9c1f27e1d764132f027948096dc8902f2b3cf898f1e6a18072627826fb505085"} Oct 30 11:11:34.512117 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.512014 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" event={"ID":"073dc996-75bd-41d8-b531-93aaf2ba4b12","Type":"ContainerDied","Data":"142073b844cc022f6e0c6f06e3a3818e9720584cb5b8762715e09fc3503f2c95"} Oct 30 11:11:34.512296 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.512119 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" event={"ID":"073dc996-75bd-41d8-b531-93aaf2ba4b12","Type":"ContainerDied","Data":"bb4657afdd237952f25127416e3df9bd958c4868c437255a98c7b4b4b575a96f"} Oct 30 11:11:34.512296 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.512182 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks" event={"ID":"073dc996-75bd-41d8-b531-93aaf2ba4b12","Type":"ContainerDied","Data":"ce82f2007189e6846d3f8eb10e82d7284524ba81e7dd600dff6a323006f7deb0"} Oct 30 11:11:34.514176 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.514155 4289 generic.go:356] "Generic (PLEG): container finished" podID="b4822533-f28b-4e97-959b-4e377442ddfd" containerID="b2386b3362d81b4da34ca01a5d45d94caa39807da09c80e94564bf55498256f0" exitCode=0 Oct 30 11:11:34.514176 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.514174 4289 generic.go:356] "Generic (PLEG): container finished" podID="b4822533-f28b-4e97-959b-4e377442ddfd" containerID="6a3c9d6f35ac1b0ed4912648c65edd9bac75bbcf3336d76ea90ac184f45eb826" exitCode=0 Oct 30 11:11:34.514295 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.514180 4289 generic.go:356] "Generic (PLEG): container finished" podID="b4822533-f28b-4e97-959b-4e377442ddfd" containerID="51fbc7c5dc57675c0ef2dd867a9011b13758074dc11f33c974b987186e2128d8" exitCode=0 Oct 30 11:11:34.514295 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.514194 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" event={"ID":"b4822533-f28b-4e97-959b-4e377442ddfd","Type":"ContainerDied","Data":"b2386b3362d81b4da34ca01a5d45d94caa39807da09c80e94564bf55498256f0"} Oct 30 11:11:34.514295 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.514221 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" Oct 30 11:11:34.514295 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.514239 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" event={"ID":"b4822533-f28b-4e97-959b-4e377442ddfd","Type":"ContainerDied","Data":"6a3c9d6f35ac1b0ed4912648c65edd9bac75bbcf3336d76ea90ac184f45eb826"} Oct 30 11:11:34.514295 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.514274 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" event={"ID":"b4822533-f28b-4e97-959b-4e377442ddfd","Type":"ContainerDied","Data":"51fbc7c5dc57675c0ef2dd867a9011b13758074dc11f33c974b987186e2128d8"} Oct 30 11:11:34.514497 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.514303 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9" event={"ID":"b4822533-f28b-4e97-959b-4e377442ddfd","Type":"ContainerDied","Data":"4b0a0400db1bd4d6ed89214120fbfc5b42aa0d2e6da35614ba3068f01fbbdd4b"} Oct 30 11:11:34.520887 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.520865 4289 scope.go:122] "RemoveContainer" containerID="ae6a182049bc84539c34da26bceed11b915379a0ca77c23f8bd562000d04d8aa" Oct 30 11:11:34.521145 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:11:34.521126 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae6a182049bc84539c34da26bceed11b915379a0ca77c23f8bd562000d04d8aa\": container with ID starting with ae6a182049bc84539c34da26bceed11b915379a0ca77c23f8bd562000d04d8aa not found: ID does not exist" containerID="ae6a182049bc84539c34da26bceed11b915379a0ca77c23f8bd562000d04d8aa" Oct 30 11:11:34.521201 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.521149 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae6a182049bc84539c34da26bceed11b915379a0ca77c23f8bd562000d04d8aa"} err="failed to get container status \"ae6a182049bc84539c34da26bceed11b915379a0ca77c23f8bd562000d04d8aa\": rpc error: code = NotFound desc = could not find container \"ae6a182049bc84539c34da26bceed11b915379a0ca77c23f8bd562000d04d8aa\": container with ID starting with ae6a182049bc84539c34da26bceed11b915379a0ca77c23f8bd562000d04d8aa not found: ID does not exist" Oct 30 11:11:34.521201 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.521173 4289 scope.go:122] "RemoveContainer" containerID="9c1f27e1d764132f027948096dc8902f2b3cf898f1e6a18072627826fb505085" Oct 30 11:11:34.529344 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.529329 4289 scope.go:122] "RemoveContainer" containerID="142073b844cc022f6e0c6f06e3a3818e9720584cb5b8762715e09fc3503f2c95" Oct 30 11:11:34.536162 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.536146 4289 scope.go:122] "RemoveContainer" containerID="bb4657afdd237952f25127416e3df9bd958c4868c437255a98c7b4b4b575a96f" Oct 30 11:11:34.543293 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.543271 4289 scope.go:122] "RemoveContainer" containerID="9c1f27e1d764132f027948096dc8902f2b3cf898f1e6a18072627826fb505085" Oct 30 11:11:34.543563 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:11:34.543544 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c1f27e1d764132f027948096dc8902f2b3cf898f1e6a18072627826fb505085\": container with ID starting with 9c1f27e1d764132f027948096dc8902f2b3cf898f1e6a18072627826fb505085 not found: ID does not exist" containerID="9c1f27e1d764132f027948096dc8902f2b3cf898f1e6a18072627826fb505085" Oct 30 11:11:34.543622 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.543569 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c1f27e1d764132f027948096dc8902f2b3cf898f1e6a18072627826fb505085"} err="failed to get container status \"9c1f27e1d764132f027948096dc8902f2b3cf898f1e6a18072627826fb505085\": rpc error: code = NotFound desc = could not find container \"9c1f27e1d764132f027948096dc8902f2b3cf898f1e6a18072627826fb505085\": container with ID starting with 9c1f27e1d764132f027948096dc8902f2b3cf898f1e6a18072627826fb505085 not found: ID does not exist" Oct 30 11:11:34.543622 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.543585 4289 scope.go:122] "RemoveContainer" containerID="142073b844cc022f6e0c6f06e3a3818e9720584cb5b8762715e09fc3503f2c95" Oct 30 11:11:34.543867 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:11:34.543840 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"142073b844cc022f6e0c6f06e3a3818e9720584cb5b8762715e09fc3503f2c95\": container with ID starting with 142073b844cc022f6e0c6f06e3a3818e9720584cb5b8762715e09fc3503f2c95 not found: ID does not exist" containerID="142073b844cc022f6e0c6f06e3a3818e9720584cb5b8762715e09fc3503f2c95" Oct 30 11:11:34.543917 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.543873 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"142073b844cc022f6e0c6f06e3a3818e9720584cb5b8762715e09fc3503f2c95"} err="failed to get container status \"142073b844cc022f6e0c6f06e3a3818e9720584cb5b8762715e09fc3503f2c95\": rpc error: code = NotFound desc = could not find container \"142073b844cc022f6e0c6f06e3a3818e9720584cb5b8762715e09fc3503f2c95\": container with ID starting with 142073b844cc022f6e0c6f06e3a3818e9720584cb5b8762715e09fc3503f2c95 not found: ID does not exist" Oct 30 11:11:34.543917 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.543888 4289 scope.go:122] "RemoveContainer" containerID="bb4657afdd237952f25127416e3df9bd958c4868c437255a98c7b4b4b575a96f" Oct 30 11:11:34.544122 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:11:34.544105 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb4657afdd237952f25127416e3df9bd958c4868c437255a98c7b4b4b575a96f\": container with ID starting with bb4657afdd237952f25127416e3df9bd958c4868c437255a98c7b4b4b575a96f not found: ID does not exist" containerID="bb4657afdd237952f25127416e3df9bd958c4868c437255a98c7b4b4b575a96f" Oct 30 11:11:34.544185 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.544130 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb4657afdd237952f25127416e3df9bd958c4868c437255a98c7b4b4b575a96f"} err="failed to get container status \"bb4657afdd237952f25127416e3df9bd958c4868c437255a98c7b4b4b575a96f\": rpc error: code = NotFound desc = could not find container \"bb4657afdd237952f25127416e3df9bd958c4868c437255a98c7b4b4b575a96f\": container with ID starting with bb4657afdd237952f25127416e3df9bd958c4868c437255a98c7b4b4b575a96f not found: ID does not exist" Oct 30 11:11:34.544185 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.544152 4289 scope.go:122] "RemoveContainer" containerID="9c1f27e1d764132f027948096dc8902f2b3cf898f1e6a18072627826fb505085" Oct 30 11:11:34.544400 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.544382 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c1f27e1d764132f027948096dc8902f2b3cf898f1e6a18072627826fb505085"} err="failed to get container status \"9c1f27e1d764132f027948096dc8902f2b3cf898f1e6a18072627826fb505085\": rpc error: code = NotFound desc = could not find container \"9c1f27e1d764132f027948096dc8902f2b3cf898f1e6a18072627826fb505085\": container with ID starting with 9c1f27e1d764132f027948096dc8902f2b3cf898f1e6a18072627826fb505085 not found: ID does not exist" Oct 30 11:11:34.544501 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.544401 4289 scope.go:122] "RemoveContainer" containerID="142073b844cc022f6e0c6f06e3a3818e9720584cb5b8762715e09fc3503f2c95" Oct 30 11:11:34.544694 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.544672 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"142073b844cc022f6e0c6f06e3a3818e9720584cb5b8762715e09fc3503f2c95"} err="failed to get container status \"142073b844cc022f6e0c6f06e3a3818e9720584cb5b8762715e09fc3503f2c95\": rpc error: code = NotFound desc = could not find container \"142073b844cc022f6e0c6f06e3a3818e9720584cb5b8762715e09fc3503f2c95\": container with ID starting with 142073b844cc022f6e0c6f06e3a3818e9720584cb5b8762715e09fc3503f2c95 not found: ID does not exist" Oct 30 11:11:34.544767 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.544695 4289 scope.go:122] "RemoveContainer" containerID="bb4657afdd237952f25127416e3df9bd958c4868c437255a98c7b4b4b575a96f" Oct 30 11:11:34.544944 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.544922 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb4657afdd237952f25127416e3df9bd958c4868c437255a98c7b4b4b575a96f"} err="failed to get container status \"bb4657afdd237952f25127416e3df9bd958c4868c437255a98c7b4b4b575a96f\": rpc error: code = NotFound desc = could not find container \"bb4657afdd237952f25127416e3df9bd958c4868c437255a98c7b4b4b575a96f\": container with ID starting with bb4657afdd237952f25127416e3df9bd958c4868c437255a98c7b4b4b575a96f not found: ID does not exist" Oct 30 11:11:34.545001 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.544946 4289 scope.go:122] "RemoveContainer" containerID="9c1f27e1d764132f027948096dc8902f2b3cf898f1e6a18072627826fb505085" Oct 30 11:11:34.545175 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.545155 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c1f27e1d764132f027948096dc8902f2b3cf898f1e6a18072627826fb505085"} err="failed to get container status \"9c1f27e1d764132f027948096dc8902f2b3cf898f1e6a18072627826fb505085\": rpc error: code = NotFound desc = could not find container \"9c1f27e1d764132f027948096dc8902f2b3cf898f1e6a18072627826fb505085\": container with ID starting with 9c1f27e1d764132f027948096dc8902f2b3cf898f1e6a18072627826fb505085 not found: ID does not exist" Oct 30 11:11:34.545175 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.545173 4289 scope.go:122] "RemoveContainer" containerID="142073b844cc022f6e0c6f06e3a3818e9720584cb5b8762715e09fc3503f2c95" Oct 30 11:11:34.545422 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.545401 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"142073b844cc022f6e0c6f06e3a3818e9720584cb5b8762715e09fc3503f2c95"} err="failed to get container status \"142073b844cc022f6e0c6f06e3a3818e9720584cb5b8762715e09fc3503f2c95\": rpc error: code = NotFound desc = could not find container \"142073b844cc022f6e0c6f06e3a3818e9720584cb5b8762715e09fc3503f2c95\": container with ID starting with 142073b844cc022f6e0c6f06e3a3818e9720584cb5b8762715e09fc3503f2c95 not found: ID does not exist" Oct 30 11:11:34.545488 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.545423 4289 scope.go:122] "RemoveContainer" containerID="bb4657afdd237952f25127416e3df9bd958c4868c437255a98c7b4b4b575a96f" Oct 30 11:11:34.545674 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.545655 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb4657afdd237952f25127416e3df9bd958c4868c437255a98c7b4b4b575a96f"} err="failed to get container status \"bb4657afdd237952f25127416e3df9bd958c4868c437255a98c7b4b4b575a96f\": rpc error: code = NotFound desc = could not find container \"bb4657afdd237952f25127416e3df9bd958c4868c437255a98c7b4b4b575a96f\": container with ID starting with bb4657afdd237952f25127416e3df9bd958c4868c437255a98c7b4b4b575a96f not found: ID does not exist" Oct 30 11:11:34.545728 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.545673 4289 scope.go:122] "RemoveContainer" containerID="b2386b3362d81b4da34ca01a5d45d94caa39807da09c80e94564bf55498256f0" Oct 30 11:11:34.552822 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.552806 4289 scope.go:122] "RemoveContainer" containerID="6a3c9d6f35ac1b0ed4912648c65edd9bac75bbcf3336d76ea90ac184f45eb826" Oct 30 11:11:34.559559 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.559541 4289 scope.go:122] "RemoveContainer" containerID="51fbc7c5dc57675c0ef2dd867a9011b13758074dc11f33c974b987186e2128d8" Oct 30 11:11:34.566349 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.566332 4289 scope.go:122] "RemoveContainer" containerID="b2386b3362d81b4da34ca01a5d45d94caa39807da09c80e94564bf55498256f0" Oct 30 11:11:34.566610 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:11:34.566593 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2386b3362d81b4da34ca01a5d45d94caa39807da09c80e94564bf55498256f0\": container with ID starting with b2386b3362d81b4da34ca01a5d45d94caa39807da09c80e94564bf55498256f0 not found: ID does not exist" containerID="b2386b3362d81b4da34ca01a5d45d94caa39807da09c80e94564bf55498256f0" Oct 30 11:11:34.566656 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.566618 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2386b3362d81b4da34ca01a5d45d94caa39807da09c80e94564bf55498256f0"} err="failed to get container status \"b2386b3362d81b4da34ca01a5d45d94caa39807da09c80e94564bf55498256f0\": rpc error: code = NotFound desc = could not find container \"b2386b3362d81b4da34ca01a5d45d94caa39807da09c80e94564bf55498256f0\": container with ID starting with b2386b3362d81b4da34ca01a5d45d94caa39807da09c80e94564bf55498256f0 not found: ID does not exist" Oct 30 11:11:34.566656 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.566639 4289 scope.go:122] "RemoveContainer" containerID="6a3c9d6f35ac1b0ed4912648c65edd9bac75bbcf3336d76ea90ac184f45eb826" Oct 30 11:11:34.566886 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:11:34.566868 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a3c9d6f35ac1b0ed4912648c65edd9bac75bbcf3336d76ea90ac184f45eb826\": container with ID starting with 6a3c9d6f35ac1b0ed4912648c65edd9bac75bbcf3336d76ea90ac184f45eb826 not found: ID does not exist" containerID="6a3c9d6f35ac1b0ed4912648c65edd9bac75bbcf3336d76ea90ac184f45eb826" Oct 30 11:11:34.566932 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.566899 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a3c9d6f35ac1b0ed4912648c65edd9bac75bbcf3336d76ea90ac184f45eb826"} err="failed to get container status \"6a3c9d6f35ac1b0ed4912648c65edd9bac75bbcf3336d76ea90ac184f45eb826\": rpc error: code = NotFound desc = could not find container \"6a3c9d6f35ac1b0ed4912648c65edd9bac75bbcf3336d76ea90ac184f45eb826\": container with ID starting with 6a3c9d6f35ac1b0ed4912648c65edd9bac75bbcf3336d76ea90ac184f45eb826 not found: ID does not exist" Oct 30 11:11:34.566932 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.566913 4289 scope.go:122] "RemoveContainer" containerID="51fbc7c5dc57675c0ef2dd867a9011b13758074dc11f33c974b987186e2128d8" Oct 30 11:11:34.567146 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:11:34.567127 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51fbc7c5dc57675c0ef2dd867a9011b13758074dc11f33c974b987186e2128d8\": container with ID starting with 51fbc7c5dc57675c0ef2dd867a9011b13758074dc11f33c974b987186e2128d8 not found: ID does not exist" containerID="51fbc7c5dc57675c0ef2dd867a9011b13758074dc11f33c974b987186e2128d8" Oct 30 11:11:34.567200 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.567150 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51fbc7c5dc57675c0ef2dd867a9011b13758074dc11f33c974b987186e2128d8"} err="failed to get container status \"51fbc7c5dc57675c0ef2dd867a9011b13758074dc11f33c974b987186e2128d8\": rpc error: code = NotFound desc = could not find container \"51fbc7c5dc57675c0ef2dd867a9011b13758074dc11f33c974b987186e2128d8\": container with ID starting with 51fbc7c5dc57675c0ef2dd867a9011b13758074dc11f33c974b987186e2128d8 not found: ID does not exist" Oct 30 11:11:34.567200 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.567166 4289 scope.go:122] "RemoveContainer" containerID="b2386b3362d81b4da34ca01a5d45d94caa39807da09c80e94564bf55498256f0" Oct 30 11:11:34.567409 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.567392 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2386b3362d81b4da34ca01a5d45d94caa39807da09c80e94564bf55498256f0"} err="failed to get container status \"b2386b3362d81b4da34ca01a5d45d94caa39807da09c80e94564bf55498256f0\": rpc error: code = NotFound desc = could not find container \"b2386b3362d81b4da34ca01a5d45d94caa39807da09c80e94564bf55498256f0\": container with ID starting with b2386b3362d81b4da34ca01a5d45d94caa39807da09c80e94564bf55498256f0 not found: ID does not exist" Oct 30 11:11:34.567478 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.567410 4289 scope.go:122] "RemoveContainer" containerID="6a3c9d6f35ac1b0ed4912648c65edd9bac75bbcf3336d76ea90ac184f45eb826" Oct 30 11:11:34.567713 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.567696 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a3c9d6f35ac1b0ed4912648c65edd9bac75bbcf3336d76ea90ac184f45eb826"} err="failed to get container status \"6a3c9d6f35ac1b0ed4912648c65edd9bac75bbcf3336d76ea90ac184f45eb826\": rpc error: code = NotFound desc = could not find container \"6a3c9d6f35ac1b0ed4912648c65edd9bac75bbcf3336d76ea90ac184f45eb826\": container with ID starting with 6a3c9d6f35ac1b0ed4912648c65edd9bac75bbcf3336d76ea90ac184f45eb826 not found: ID does not exist" Oct 30 11:11:34.567713 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.567712 4289 scope.go:122] "RemoveContainer" containerID="51fbc7c5dc57675c0ef2dd867a9011b13758074dc11f33c974b987186e2128d8" Oct 30 11:11:34.567933 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.567914 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51fbc7c5dc57675c0ef2dd867a9011b13758074dc11f33c974b987186e2128d8"} err="failed to get container status \"51fbc7c5dc57675c0ef2dd867a9011b13758074dc11f33c974b987186e2128d8\": rpc error: code = NotFound desc = could not find container \"51fbc7c5dc57675c0ef2dd867a9011b13758074dc11f33c974b987186e2128d8\": container with ID starting with 51fbc7c5dc57675c0ef2dd867a9011b13758074dc11f33c974b987186e2128d8 not found: ID does not exist" Oct 30 11:11:34.567998 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.567934 4289 scope.go:122] "RemoveContainer" containerID="b2386b3362d81b4da34ca01a5d45d94caa39807da09c80e94564bf55498256f0" Oct 30 11:11:34.568146 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.568129 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2386b3362d81b4da34ca01a5d45d94caa39807da09c80e94564bf55498256f0"} err="failed to get container status \"b2386b3362d81b4da34ca01a5d45d94caa39807da09c80e94564bf55498256f0\": rpc error: code = NotFound desc = could not find container \"b2386b3362d81b4da34ca01a5d45d94caa39807da09c80e94564bf55498256f0\": container with ID starting with b2386b3362d81b4da34ca01a5d45d94caa39807da09c80e94564bf55498256f0 not found: ID does not exist" Oct 30 11:11:34.568146 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.568145 4289 scope.go:122] "RemoveContainer" containerID="6a3c9d6f35ac1b0ed4912648c65edd9bac75bbcf3336d76ea90ac184f45eb826" Oct 30 11:11:34.568346 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.568327 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a3c9d6f35ac1b0ed4912648c65edd9bac75bbcf3336d76ea90ac184f45eb826"} err="failed to get container status \"6a3c9d6f35ac1b0ed4912648c65edd9bac75bbcf3336d76ea90ac184f45eb826\": rpc error: code = NotFound desc = could not find container \"6a3c9d6f35ac1b0ed4912648c65edd9bac75bbcf3336d76ea90ac184f45eb826\": container with ID starting with 6a3c9d6f35ac1b0ed4912648c65edd9bac75bbcf3336d76ea90ac184f45eb826 not found: ID does not exist" Oct 30 11:11:34.568395 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.568347 4289 scope.go:122] "RemoveContainer" containerID="51fbc7c5dc57675c0ef2dd867a9011b13758074dc11f33c974b987186e2128d8" Oct 30 11:11:34.568742 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.568724 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51fbc7c5dc57675c0ef2dd867a9011b13758074dc11f33c974b987186e2128d8"} err="failed to get container status \"51fbc7c5dc57675c0ef2dd867a9011b13758074dc11f33c974b987186e2128d8\": rpc error: code = NotFound desc = could not find container \"51fbc7c5dc57675c0ef2dd867a9011b13758074dc11f33c974b987186e2128d8\": container with ID starting with 51fbc7c5dc57675c0ef2dd867a9011b13758074dc11f33c974b987186e2128d8 not found: ID does not exist" Oct 30 11:11:34.579530 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.579506 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8"] Oct 30 11:11:34.582549 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.582531 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-pvcx8"] Oct 30 11:11:34.606419 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.606394 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:34.630244 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.630186 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8"] Oct 30 11:11:34.631602 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.631585 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m6wz8"] Oct 30 11:11:34.671722 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.671703 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9"] Oct 30 11:11:34.674378 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.674356 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-x8zb9"] Oct 30 11:11:34.711620 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.711595 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks"] Oct 30 11:11:34.716071 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.716051 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-t5tks"] Oct 30 11:11:34.844156 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.844111 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:11:34.844415 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.844374 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="845b7403-ad65-448e-97d4-6fb29627e099" containerName="otc-container" containerID="cri-o://b8abbf1f8aca8eab9fb7ea6f4cbfab39e2212f497bb91472bb803b41addbaa76" gracePeriod=30 Oct 30 11:11:34.862302 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.862269 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:34.973553 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:34.973530 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:11:35.034466 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.034395 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-kube-api-access-4nkpq\" (UniqueName: \"kubernetes.io/projected/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-kube-api-access-4nkpq\") pod \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " Oct 30 11:11:35.034643 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.034482 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-tls-assets\" (UniqueName: \"kubernetes.io/projected/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-tls-assets\") pod \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " Oct 30 11:11:35.034643 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.034524 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " Oct 30 11:11:35.034643 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.034573 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-thanos-prometheus-http-client-file\") pod \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " Oct 30 11:11:35.034643 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.034609 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-config-out\" (UniqueName: \"kubernetes.io/empty-dir/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-config-out\") pod \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " Oct 30 11:11:35.034869 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.034654 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " Oct 30 11:11:35.034869 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.034686 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-secret-prometheus-web-tls-ca\") pod \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " Oct 30 11:11:35.034869 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.034726 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " Oct 30 11:11:35.034869 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.034757 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " Oct 30 11:11:35.034869 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.034799 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-config\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-config\") pod \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " Oct 30 11:11:35.034869 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.034836 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " Oct 30 11:11:35.035162 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.034896 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-web-config\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-web-config\") pod \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " Oct 30 11:11:35.035162 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.035032 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\" (UID: \"8ab3b83f-baa8-447b-ad20-76c66c15a7f8\") " Oct 30 11:11:35.036986 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.036929 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-prometheus-data-science-monitoringstack-rulefiles-0" pod "8ab3b83f-baa8-447b-ad20-76c66c15a7f8" (UID: "8ab3b83f-baa8-447b-ad20-76c66c15a7f8"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:11:35.037386 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.037348 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-prometheus-data-science-monitoringstack-rulefiles-1" pod "8ab3b83f-baa8-447b-ad20-76c66c15a7f8" (UID: "8ab3b83f-baa8-447b-ad20-76c66c15a7f8"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:11:35.037591 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.037554 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-prometheus-data-science-monitoringstack-rulefiles-2" pod "8ab3b83f-baa8-447b-ad20-76c66c15a7f8" (UID: "8ab3b83f-baa8-447b-ad20-76c66c15a7f8"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:11:35.039139 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.039112 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "8ab3b83f-baa8-447b-ad20-76c66c15a7f8" (UID: "8ab3b83f-baa8-447b-ad20-76c66c15a7f8"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:11:35.040306 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.040277 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "8ab3b83f-baa8-447b-ad20-76c66c15a7f8" (UID: "8ab3b83f-baa8-447b-ad20-76c66c15a7f8"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:11:35.040406 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.040354 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-thanos-prometheus-http-client-file" pod "8ab3b83f-baa8-447b-ad20-76c66c15a7f8" (UID: "8ab3b83f-baa8-447b-ad20-76c66c15a7f8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:11:35.041553 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.041519 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-kube-api-access-4nkpq" pod "8ab3b83f-baa8-447b-ad20-76c66c15a7f8" (UID: "8ab3b83f-baa8-447b-ad20-76c66c15a7f8"). InnerVolumeSpecName "kube-api-access-4nkpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:11:35.041759 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.041662 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-secret-prometheus-web-tls-ca" pod "8ab3b83f-baa8-447b-ad20-76c66c15a7f8" (UID: "8ab3b83f-baa8-447b-ad20-76c66c15a7f8"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:11:35.041759 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.041730 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-config-out" pod "8ab3b83f-baa8-447b-ad20-76c66c15a7f8" (UID: "8ab3b83f-baa8-447b-ad20-76c66c15a7f8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:11:35.041864 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.041793 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-config" pod "8ab3b83f-baa8-447b-ad20-76c66c15a7f8" (UID: "8ab3b83f-baa8-447b-ad20-76c66c15a7f8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:11:35.042194 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.042159 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48" pod "8ab3b83f-baa8-447b-ad20-76c66c15a7f8" (UID: "8ab3b83f-baa8-447b-ad20-76c66c15a7f8"). InnerVolumeSpecName "pvc-c799cc48-125a-421c-8fc6-8a4d017a7653". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 11:11:35.042264 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.042192 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-tls-assets" pod "8ab3b83f-baa8-447b-ad20-76c66c15a7f8" (UID: "8ab3b83f-baa8-447b-ad20-76c66c15a7f8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:11:35.046936 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.046912 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:35.057063 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.057040 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-web-config" pod "8ab3b83f-baa8-447b-ad20-76c66c15a7f8" (UID: "8ab3b83f-baa8-447b-ad20-76c66c15a7f8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:11:35.088092 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.088068 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:11:35.135871 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.135841 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/845b7403-ad65-448e-97d4-6fb29627e099-otc-internal\" (UniqueName: \"kubernetes.io/configmap/845b7403-ad65-448e-97d4-6fb29627e099-otc-internal\") pod \"845b7403-ad65-448e-97d4-6fb29627e099\" (UID: \"845b7403-ad65-448e-97d4-6fb29627e099\") " Oct 30 11:11:35.136000 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.135894 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/845b7403-ad65-448e-97d4-6fb29627e099-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/845b7403-ad65-448e-97d4-6fb29627e099-monitoring-tls-certs\") pod \"845b7403-ad65-448e-97d4-6fb29627e099\" (UID: \"845b7403-ad65-448e-97d4-6fb29627e099\") " Oct 30 11:11:35.136000 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.135933 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/845b7403-ad65-448e-97d4-6fb29627e099-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/845b7403-ad65-448e-97d4-6fb29627e099-data-science-collector-ta-client-cert\") pod \"845b7403-ad65-448e-97d4-6fb29627e099\" (UID: \"845b7403-ad65-448e-97d4-6fb29627e099\") " Oct 30 11:11:35.136000 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.135999 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/845b7403-ad65-448e-97d4-6fb29627e099-tls-certs\" (UniqueName: \"kubernetes.io/secret/845b7403-ad65-448e-97d4-6fb29627e099-tls-certs\") pod \"845b7403-ad65-448e-97d4-6fb29627e099\" (UID: \"845b7403-ad65-448e-97d4-6fb29627e099\") " Oct 30 11:11:35.136185 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.136034 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/845b7403-ad65-448e-97d4-6fb29627e099-kube-api-access-kr5xm\" (UniqueName: \"kubernetes.io/projected/845b7403-ad65-448e-97d4-6fb29627e099-kube-api-access-kr5xm\") pod \"845b7403-ad65-448e-97d4-6fb29627e099\" (UID: \"845b7403-ad65-448e-97d4-6fb29627e099\") " Oct 30 11:11:35.136251 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.136200 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nkpq\" (UniqueName: \"kubernetes.io/projected/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-kube-api-access-4nkpq\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:35.136251 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.136217 4289 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-tls-assets\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:35.136251 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.136235 4289 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:35.136251 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.136238 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845b7403-ad65-448e-97d4-6fb29627e099-otc-internal" pod "845b7403-ad65-448e-97d4-6fb29627e099" (UID: "845b7403-ad65-448e-97d4-6fb29627e099"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:11:35.136491 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.136251 4289 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:35.136491 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.136268 4289 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-config-out\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:35.136491 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.136284 4289 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:35.136491 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.136298 4289 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-secret-prometheus-web-tls-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:35.136491 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.136314 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:35.136491 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.136330 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:35.136491 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.136344 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:35.136491 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.136359 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:35.136491 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.136373 4289 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8ab3b83f-baa8-447b-ad20-76c66c15a7f8-web-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:35.136491 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.136403 4289 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") on node \"ip-10-0-137-242.ec2.internal\" " Oct 30 11:11:35.140948 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.140919 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/845b7403-ad65-448e-97d4-6fb29627e099-data-science-collector-ta-client-cert" pod "845b7403-ad65-448e-97d4-6fb29627e099" (UID: "845b7403-ad65-448e-97d4-6fb29627e099"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:11:35.141252 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.140963 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/845b7403-ad65-448e-97d4-6fb29627e099-monitoring-tls-certs" pod "845b7403-ad65-448e-97d4-6fb29627e099" (UID: "845b7403-ad65-448e-97d4-6fb29627e099"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:11:35.141252 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.140980 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/845b7403-ad65-448e-97d4-6fb29627e099-tls-certs" pod "845b7403-ad65-448e-97d4-6fb29627e099" (UID: "845b7403-ad65-448e-97d4-6fb29627e099"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:11:35.142556 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.142527 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845b7403-ad65-448e-97d4-6fb29627e099-kube-api-access-kr5xm" pod "845b7403-ad65-448e-97d4-6fb29627e099" (UID: "845b7403-ad65-448e-97d4-6fb29627e099"). InnerVolumeSpecName "kube-api-access-kr5xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:11:35.149705 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.149689 4289 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-c799cc48-125a-421c-8fc6-8a4d017a7653" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48") on node "ip-10-0-137-242.ec2.internal" Oct 30 11:11:35.237384 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.237355 4289 reconciler_common.go:299] "Volume detached for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"csi-ea06c66241a0a003a5daff5d9fc602f40e53768b00b104c5e4118b92d558c090\"" Oct 30 11:11:35.237384 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.237382 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/845b7403-ad65-448e-97d4-6fb29627e099-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:35.237578 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.237392 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr5xm\" (UniqueName: \"kubernetes.io/projected/845b7403-ad65-448e-97d4-6fb29627e099-kube-api-access-kr5xm\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:35.237578 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.237401 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/845b7403-ad65-448e-97d4-6fb29627e099-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:35.237578 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.237410 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/845b7403-ad65-448e-97d4-6fb29627e099-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:35.237578 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.237421 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/845b7403-ad65-448e-97d4-6fb29627e099-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:35.523745 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.523658 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"8ab3b83f-baa8-447b-ad20-76c66c15a7f8","Type":"ContainerDied","Data":"f0c2e73867413e048ce4b8e1e7b351bac988b3766f91636f08321ce4daf83e8c"} Oct 30 11:11:35.523745 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.523737 4289 scope.go:122] "RemoveContainer" containerID="895d89ae220db3342f65c5ca22e70050b02d4140d094fccde9ed4235a7383c32" Oct 30 11:11:35.523967 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.523669 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:11:35.527009 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.526968 4289 generic.go:356] "Generic (PLEG): container finished" podID="845b7403-ad65-448e-97d4-6fb29627e099" containerID="b8abbf1f8aca8eab9fb7ea6f4cbfab39e2212f497bb91472bb803b41addbaa76" exitCode=0 Oct 30 11:11:35.527131 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.527001 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"845b7403-ad65-448e-97d4-6fb29627e099","Type":"ContainerDied","Data":"b8abbf1f8aca8eab9fb7ea6f4cbfab39e2212f497bb91472bb803b41addbaa76"} Oct 30 11:11:35.527131 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.527019 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:11:35.527131 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.527052 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"845b7403-ad65-448e-97d4-6fb29627e099","Type":"ContainerDied","Data":"f3101be966cb65d825c9a76e076b2e19a63977cd8706c9e03f76468d8d0cfcde"} Oct 30 11:11:35.534099 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.534073 4289 scope.go:122] "RemoveContainer" containerID="a4ef55cd9710a012596efb4e6c0c877bc8a32b41f27b98eb4393aa79e9593098" Oct 30 11:11:35.542327 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.542306 4289 scope.go:122] "RemoveContainer" containerID="13d58b742224e5fae291138650bc03a4f617f437fb6e647d9f7961d144fcbe13" Oct 30 11:11:35.549906 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.549886 4289 scope.go:122] "RemoveContainer" containerID="1e76ebf63bc9ac996b09650186494f379b60a9bd49b5c5697228d6bd15b6b70e" Oct 30 11:11:35.557634 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.557615 4289 scope.go:122] "RemoveContainer" containerID="b8abbf1f8aca8eab9fb7ea6f4cbfab39e2212f497bb91472bb803b41addbaa76" Oct 30 11:11:35.564684 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.564663 4289 scope.go:122] "RemoveContainer" containerID="b8abbf1f8aca8eab9fb7ea6f4cbfab39e2212f497bb91472bb803b41addbaa76" Oct 30 11:11:35.564971 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:11:35.564952 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8abbf1f8aca8eab9fb7ea6f4cbfab39e2212f497bb91472bb803b41addbaa76\": container with ID starting with b8abbf1f8aca8eab9fb7ea6f4cbfab39e2212f497bb91472bb803b41addbaa76 not found: ID does not exist" containerID="b8abbf1f8aca8eab9fb7ea6f4cbfab39e2212f497bb91472bb803b41addbaa76" Oct 30 11:11:35.565023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.564979 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8abbf1f8aca8eab9fb7ea6f4cbfab39e2212f497bb91472bb803b41addbaa76"} err="failed to get container status \"b8abbf1f8aca8eab9fb7ea6f4cbfab39e2212f497bb91472bb803b41addbaa76\": rpc error: code = NotFound desc = could not find container \"b8abbf1f8aca8eab9fb7ea6f4cbfab39e2212f497bb91472bb803b41addbaa76\": container with ID starting with b8abbf1f8aca8eab9fb7ea6f4cbfab39e2212f497bb91472bb803b41addbaa76 not found: ID does not exist" Oct 30 11:11:35.575304 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.575280 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:11:35.577524 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.577503 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:11:35.616907 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.616880 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:11:35.624587 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.624564 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="073dc996-75bd-41d8-b531-93aaf2ba4b12" path="/var/lib/kubelet/pods/073dc996-75bd-41d8-b531-93aaf2ba4b12/volumes" Oct 30 11:11:35.625092 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.625073 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a5f36fe-3925-420e-9a5f-1e2d1e70a78e" path="/var/lib/kubelet/pods/4a5f36fe-3925-420e-9a5f-1e2d1e70a78e/volumes" Oct 30 11:11:35.625496 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.625480 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="742466b8-3c11-4a49-90bd-08aae667cdaf" path="/var/lib/kubelet/pods/742466b8-3c11-4a49-90bd-08aae667cdaf/volumes" Oct 30 11:11:35.626158 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.626139 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="845b7403-ad65-448e-97d4-6fb29627e099" path="/var/lib/kubelet/pods/845b7403-ad65-448e-97d4-6fb29627e099/volumes" Oct 30 11:11:35.626551 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.626536 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4822533-f28b-4e97-959b-4e377442ddfd" path="/var/lib/kubelet/pods/b4822533-f28b-4e97-959b-4e377442ddfd/volumes" Oct 30 11:11:35.628948 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.628925 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:11:35.796266 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.796182 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:35.963558 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.963521 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="75a3f2af-e799-4bfc-bc43-f40ab46c0913" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.35:9093/-/ready\": dial tcp 10.134.0.35:9093: connect: connection refused" Oct 30 11:11:35.972670 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:35.972644 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:36.146328 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:36.146250 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:36.940339 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:36.940310 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:37.133643 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:37.133610 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:37.624404 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:37.624372 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ab3b83f-baa8-447b-ad20-76c66c15a7f8" path="/var/lib/kubelet/pods/8ab3b83f-baa8-447b-ad20-76c66c15a7f8/volumes" Oct 30 11:11:38.056168 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056138 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-m8ln4"] Oct 30 11:11:38.056555 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056474 4289 cpu_manager.go:414] "containerMap: removing container" podUID="073dc996-75bd-41d8-b531-93aaf2ba4b12" containerName="method-gate" Oct 30 11:11:38.056555 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056485 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073dc996-75bd-41d8-b531-93aaf2ba4b12" containerName="method-gate" Oct 30 11:11:38.056555 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056494 4289 cpu_manager.go:414] "containerMap: removing container" podUID="8ab3b83f-baa8-447b-ad20-76c66c15a7f8" containerName="init-config-reloader" Oct 30 11:11:38.056555 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056499 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab3b83f-baa8-447b-ad20-76c66c15a7f8" containerName="init-config-reloader" Oct 30 11:11:38.056555 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056508 4289 cpu_manager.go:414] "containerMap: removing container" podUID="8ab3b83f-baa8-447b-ad20-76c66c15a7f8" containerName="prometheus" Oct 30 11:11:38.056555 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056514 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab3b83f-baa8-447b-ad20-76c66c15a7f8" containerName="prometheus" Oct 30 11:11:38.056555 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056522 4289 cpu_manager.go:414] "containerMap: removing container" podUID="4a5f36fe-3925-420e-9a5f-1e2d1e70a78e" containerName="korrel8r" Oct 30 11:11:38.056555 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056526 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a5f36fe-3925-420e-9a5f-1e2d1e70a78e" containerName="korrel8r" Oct 30 11:11:38.056555 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056533 4289 cpu_manager.go:414] "containerMap: removing container" podUID="8ab3b83f-baa8-447b-ad20-76c66c15a7f8" containerName="thanos-sidecar" Oct 30 11:11:38.056555 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056537 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab3b83f-baa8-447b-ad20-76c66c15a7f8" containerName="thanos-sidecar" Oct 30 11:11:38.056555 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056546 4289 cpu_manager.go:414] "containerMap: removing container" podUID="b4822533-f28b-4e97-959b-4e377442ddfd" containerName="kube-rbac-proxy" Oct 30 11:11:38.056555 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056550 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4822533-f28b-4e97-959b-4e377442ddfd" containerName="kube-rbac-proxy" Oct 30 11:11:38.056555 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056556 4289 cpu_manager.go:414] "containerMap: removing container" podUID="8ab3b83f-baa8-447b-ad20-76c66c15a7f8" containerName="config-reloader" Oct 30 11:11:38.056555 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056561 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab3b83f-baa8-447b-ad20-76c66c15a7f8" containerName="config-reloader" Oct 30 11:11:38.056954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056567 4289 cpu_manager.go:414] "containerMap: removing container" podUID="073dc996-75bd-41d8-b531-93aaf2ba4b12" containerName="prom-label-proxy" Oct 30 11:11:38.056954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056572 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073dc996-75bd-41d8-b531-93aaf2ba4b12" containerName="prom-label-proxy" Oct 30 11:11:38.056954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056584 4289 cpu_manager.go:414] "containerMap: removing container" podUID="073dc996-75bd-41d8-b531-93aaf2ba4b12" containerName="kube-rbac-proxy" Oct 30 11:11:38.056954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056589 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073dc996-75bd-41d8-b531-93aaf2ba4b12" containerName="kube-rbac-proxy" Oct 30 11:11:38.056954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056597 4289 cpu_manager.go:414] "containerMap: removing container" podUID="742466b8-3c11-4a49-90bd-08aae667cdaf" containerName="kube-rbac-proxy" Oct 30 11:11:38.056954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056601 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="742466b8-3c11-4a49-90bd-08aae667cdaf" containerName="kube-rbac-proxy" Oct 30 11:11:38.056954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056610 4289 cpu_manager.go:414] "containerMap: removing container" podUID="b4822533-f28b-4e97-959b-4e377442ddfd" containerName="prom-label-proxy" Oct 30 11:11:38.056954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056615 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4822533-f28b-4e97-959b-4e377442ddfd" containerName="prom-label-proxy" Oct 30 11:11:38.056954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056621 4289 cpu_manager.go:414] "containerMap: removing container" podUID="845b7403-ad65-448e-97d4-6fb29627e099" containerName="otc-container" Oct 30 11:11:38.056954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056625 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845b7403-ad65-448e-97d4-6fb29627e099" containerName="otc-container" Oct 30 11:11:38.056954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056634 4289 cpu_manager.go:414] "containerMap: removing container" podUID="b4822533-f28b-4e97-959b-4e377442ddfd" containerName="method-gate" Oct 30 11:11:38.056954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056637 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4822533-f28b-4e97-959b-4e377442ddfd" containerName="method-gate" Oct 30 11:11:38.056954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056683 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4822533-f28b-4e97-959b-4e377442ddfd" containerName="prom-label-proxy" Oct 30 11:11:38.056954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056690 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="845b7403-ad65-448e-97d4-6fb29627e099" containerName="otc-container" Oct 30 11:11:38.056954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056697 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4822533-f28b-4e97-959b-4e377442ddfd" containerName="method-gate" Oct 30 11:11:38.056954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056704 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="073dc996-75bd-41d8-b531-93aaf2ba4b12" containerName="method-gate" Oct 30 11:11:38.056954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056710 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab3b83f-baa8-447b-ad20-76c66c15a7f8" containerName="prometheus" Oct 30 11:11:38.056954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056715 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a5f36fe-3925-420e-9a5f-1e2d1e70a78e" containerName="korrel8r" Oct 30 11:11:38.056954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056720 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab3b83f-baa8-447b-ad20-76c66c15a7f8" containerName="thanos-sidecar" Oct 30 11:11:38.056954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056726 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4822533-f28b-4e97-959b-4e377442ddfd" containerName="kube-rbac-proxy" Oct 30 11:11:38.056954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056730 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab3b83f-baa8-447b-ad20-76c66c15a7f8" containerName="config-reloader" Oct 30 11:11:38.056954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056736 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="073dc996-75bd-41d8-b531-93aaf2ba4b12" containerName="prom-label-proxy" Oct 30 11:11:38.056954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056742 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="073dc996-75bd-41d8-b531-93aaf2ba4b12" containerName="kube-rbac-proxy" Oct 30 11:11:38.056954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.056747 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="742466b8-3c11-4a49-90bd-08aae667cdaf" containerName="kube-rbac-proxy" Oct 30 11:11:38.061256 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.061236 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-m8ln4" Oct 30 11:11:38.064985 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.064960 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:38.084244 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.084219 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-m8ln4"] Oct 30 11:11:38.094694 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.094674 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 11:11:38.095859 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.095833 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 11:11:38.095859 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.095836 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-jkmrr\"" Oct 30 11:11:38.161890 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.161862 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpqgr\" (UniqueName: \"kubernetes.io/projected/29a657d4-73ed-479b-88e5-660e99384273-kube-api-access-mpqgr\") pod \"korrel8r-6644769df5-m8ln4\" (UID: \"29a657d4-73ed-479b-88e5-660e99384273\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-m8ln4" Oct 30 11:11:38.162063 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.161893 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/29a657d4-73ed-479b-88e5-660e99384273-tls\") pod \"korrel8r-6644769df5-m8ln4\" (UID: \"29a657d4-73ed-479b-88e5-660e99384273\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-m8ln4" Oct 30 11:11:38.162063 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.161927 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/29a657d4-73ed-479b-88e5-660e99384273-config\") pod \"korrel8r-6644769df5-m8ln4\" (UID: \"29a657d4-73ed-479b-88e5-660e99384273\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-m8ln4" Oct 30 11:11:38.242474 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.242419 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:38.262389 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.262361 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/29a657d4-73ed-479b-88e5-660e99384273-config\") pod \"korrel8r-6644769df5-m8ln4\" (UID: \"29a657d4-73ed-479b-88e5-660e99384273\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-m8ln4" Oct 30 11:11:38.262528 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.262423 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpqgr\" (UniqueName: \"kubernetes.io/projected/29a657d4-73ed-479b-88e5-660e99384273-kube-api-access-mpqgr\") pod \"korrel8r-6644769df5-m8ln4\" (UID: \"29a657d4-73ed-479b-88e5-660e99384273\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-m8ln4" Oct 30 11:11:38.262528 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.262475 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/29a657d4-73ed-479b-88e5-660e99384273-tls\") pod \"korrel8r-6644769df5-m8ln4\" (UID: \"29a657d4-73ed-479b-88e5-660e99384273\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-m8ln4" Oct 30 11:11:38.263174 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.263152 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/29a657d4-73ed-479b-88e5-660e99384273-config\") pod \"korrel8r-6644769df5-m8ln4\" (UID: \"29a657d4-73ed-479b-88e5-660e99384273\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-m8ln4" Oct 30 11:11:38.264817 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.264795 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/29a657d4-73ed-479b-88e5-660e99384273-tls\") pod \"korrel8r-6644769df5-m8ln4\" (UID: \"29a657d4-73ed-479b-88e5-660e99384273\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-m8ln4" Oct 30 11:11:38.285536 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.285510 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpqgr\" (UniqueName: \"kubernetes.io/projected/29a657d4-73ed-479b-88e5-660e99384273-kube-api-access-mpqgr\") pod \"korrel8r-6644769df5-m8ln4\" (UID: \"29a657d4-73ed-479b-88e5-660e99384273\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-m8ln4" Oct 30 11:11:38.379495 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.379412 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-m8ln4" Oct 30 11:11:38.421975 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.421938 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:38.676288 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.676258 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-m8ln4"] Oct 30 11:11:38.676407 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:11:38.676344 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a657d4_73ed_479b_88e5_660e99384273.slice/crio-4defc040a5f748b8e2b085ec52655d8179e83ec37a1b3498f8dd1e17b3d122cc WatchSource:0}: Error finding container 4defc040a5f748b8e2b085ec52655d8179e83ec37a1b3498f8dd1e17b3d122cc: Status 404 returned error can't find the container with id 4defc040a5f748b8e2b085ec52655d8179e83ec37a1b3498f8dd1e17b3d122cc Oct 30 11:11:38.851400 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:38.851355 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:39.014765 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:39.014730 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:39.493045 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:39.493008 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:39.549460 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:39.549414 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-m8ln4" event={"ID":"29a657d4-73ed-479b-88e5-660e99384273","Type":"ContainerStarted","Data":"b3254c7a9593b972aab515bdd08486d6598d65793f784c34393c92a5f1cd5a63"} Oct 30 11:11:39.549603 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:39.549477 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-m8ln4" event={"ID":"29a657d4-73ed-479b-88e5-660e99384273","Type":"ContainerStarted","Data":"4defc040a5f748b8e2b085ec52655d8179e83ec37a1b3498f8dd1e17b3d122cc"} Oct 30 11:11:39.549603 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:39.549571 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-m8ln4" Oct 30 11:11:39.590163 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:39.590119 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-m8ln4" podStartSLOduration=1.590105962 podStartE2EDuration="1.590105962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:11:39.588822558 +0000 UTC m=+1590.090202065" watchObservedRunningTime="2026-10-30 11:11:39.590105962 +0000 UTC m=+1590.091485446" Oct 30 11:11:39.656033 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:39.655995 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:11:40.963119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:40.963084 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="75a3f2af-e799-4bfc-bc43-f40ab46c0913" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.35:9093/-/ready\": dial tcp 10.134.0.35:9093: connect: connection refused" Oct 30 11:11:44.389513 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.389487 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:11:44.512677 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.512650 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75a3f2af-e799-4bfc-bc43-f40ab46c0913-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75a3f2af-e799-4bfc-bc43-f40ab46c0913-cluster-tls-config\") pod \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " Oct 30 11:11:44.512840 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.512688 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75a3f2af-e799-4bfc-bc43-f40ab46c0913-config-volume\" (UniqueName: \"kubernetes.io/secret/75a3f2af-e799-4bfc-bc43-f40ab46c0913-config-volume\") pod \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " Oct 30 11:11:44.512840 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.512716 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a3f2af-e799-4bfc-bc43-f40ab46c0913-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/75a3f2af-e799-4bfc-bc43-f40ab46c0913-alertmanager-data-science-monitoringstack-db\") pod \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " Oct 30 11:11:44.512840 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.512749 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75a3f2af-e799-4bfc-bc43-f40ab46c0913-web-config\" (UniqueName: \"kubernetes.io/secret/75a3f2af-e799-4bfc-bc43-f40ab46c0913-web-config\") pod \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " Oct 30 11:11:44.512840 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.512793 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a3f2af-e799-4bfc-bc43-f40ab46c0913-tls-assets\" (UniqueName: \"kubernetes.io/projected/75a3f2af-e799-4bfc-bc43-f40ab46c0913-tls-assets\") pod \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " Oct 30 11:11:44.512840 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.512821 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a3f2af-e799-4bfc-bc43-f40ab46c0913-config-out\" (UniqueName: \"kubernetes.io/empty-dir/75a3f2af-e799-4bfc-bc43-f40ab46c0913-config-out\") pod \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " Oct 30 11:11:44.512840 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.512842 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a3f2af-e799-4bfc-bc43-f40ab46c0913-kube-api-access-dbx6p\" (UniqueName: \"kubernetes.io/projected/75a3f2af-e799-4bfc-bc43-f40ab46c0913-kube-api-access-dbx6p\") pod \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\" (UID: \"75a3f2af-e799-4bfc-bc43-f40ab46c0913\") " Oct 30 11:11:44.513169 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.513098 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a3f2af-e799-4bfc-bc43-f40ab46c0913-alertmanager-data-science-monitoringstack-db" pod "75a3f2af-e799-4bfc-bc43-f40ab46c0913" (UID: "75a3f2af-e799-4bfc-bc43-f40ab46c0913"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:11:44.514884 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.514855 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a3f2af-e799-4bfc-bc43-f40ab46c0913-kube-api-access-dbx6p" pod "75a3f2af-e799-4bfc-bc43-f40ab46c0913" (UID: "75a3f2af-e799-4bfc-bc43-f40ab46c0913"). InnerVolumeSpecName "kube-api-access-dbx6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:11:44.515005 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.514971 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a3f2af-e799-4bfc-bc43-f40ab46c0913-tls-assets" pod "75a3f2af-e799-4bfc-bc43-f40ab46c0913" (UID: "75a3f2af-e799-4bfc-bc43-f40ab46c0913"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:11:44.515270 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.515247 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a3f2af-e799-4bfc-bc43-f40ab46c0913-config-volume" pod "75a3f2af-e799-4bfc-bc43-f40ab46c0913" (UID: "75a3f2af-e799-4bfc-bc43-f40ab46c0913"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:11:44.515369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.515299 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a3f2af-e799-4bfc-bc43-f40ab46c0913-config-out" pod "75a3f2af-e799-4bfc-bc43-f40ab46c0913" (UID: "75a3f2af-e799-4bfc-bc43-f40ab46c0913"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:11:44.518861 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.518840 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a3f2af-e799-4bfc-bc43-f40ab46c0913-cluster-tls-config" pod "75a3f2af-e799-4bfc-bc43-f40ab46c0913" (UID: "75a3f2af-e799-4bfc-bc43-f40ab46c0913"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:11:44.525693 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.525667 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a3f2af-e799-4bfc-bc43-f40ab46c0913-web-config" pod "75a3f2af-e799-4bfc-bc43-f40ab46c0913" (UID: "75a3f2af-e799-4bfc-bc43-f40ab46c0913"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:11:44.573350 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.573320 4289 generic.go:356] "Generic (PLEG): container finished" podID="75a3f2af-e799-4bfc-bc43-f40ab46c0913" containerID="e02a4ab64b7d12c2be335320aed7c7677c3f3914d674e7f7a3790301c6ba5e61" exitCode=0 Oct 30 11:11:44.573464 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.573412 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"75a3f2af-e799-4bfc-bc43-f40ab46c0913","Type":"ContainerDied","Data":"e02a4ab64b7d12c2be335320aed7c7677c3f3914d674e7f7a3790301c6ba5e61"} Oct 30 11:11:44.573464 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.573457 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:11:44.573538 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.573484 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"75a3f2af-e799-4bfc-bc43-f40ab46c0913","Type":"ContainerDied","Data":"843fa932192557a70ef67f6b91568dec72185fa0750db6b4b4d0f3674cc3a7f2"} Oct 30 11:11:44.573538 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.573513 4289 scope.go:122] "RemoveContainer" containerID="f4b8bfdabfb4b4303b7e6e83d20b5006d074b9ea0cc0caa97b8bde7d898677d2" Oct 30 11:11:44.581781 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.581765 4289 scope.go:122] "RemoveContainer" containerID="e02a4ab64b7d12c2be335320aed7c7677c3f3914d674e7f7a3790301c6ba5e61" Oct 30 11:11:44.588700 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.588679 4289 scope.go:122] "RemoveContainer" containerID="77ba23d9377c25b188c49d964a594a298d1ff3366744dc2b698cc841933ca93d" Oct 30 11:11:44.595439 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.595416 4289 scope.go:122] "RemoveContainer" containerID="f4b8bfdabfb4b4303b7e6e83d20b5006d074b9ea0cc0caa97b8bde7d898677d2" Oct 30 11:11:44.595673 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:11:44.595657 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b8bfdabfb4b4303b7e6e83d20b5006d074b9ea0cc0caa97b8bde7d898677d2\": container with ID starting with f4b8bfdabfb4b4303b7e6e83d20b5006d074b9ea0cc0caa97b8bde7d898677d2 not found: ID does not exist" containerID="f4b8bfdabfb4b4303b7e6e83d20b5006d074b9ea0cc0caa97b8bde7d898677d2" Oct 30 11:11:44.595716 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.595681 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b8bfdabfb4b4303b7e6e83d20b5006d074b9ea0cc0caa97b8bde7d898677d2"} err="failed to get container status \"f4b8bfdabfb4b4303b7e6e83d20b5006d074b9ea0cc0caa97b8bde7d898677d2\": rpc error: code = NotFound desc = could not find container \"f4b8bfdabfb4b4303b7e6e83d20b5006d074b9ea0cc0caa97b8bde7d898677d2\": container with ID starting with f4b8bfdabfb4b4303b7e6e83d20b5006d074b9ea0cc0caa97b8bde7d898677d2 not found: ID does not exist" Oct 30 11:11:44.595716 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.595696 4289 scope.go:122] "RemoveContainer" containerID="e02a4ab64b7d12c2be335320aed7c7677c3f3914d674e7f7a3790301c6ba5e61" Oct 30 11:11:44.595952 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:11:44.595934 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e02a4ab64b7d12c2be335320aed7c7677c3f3914d674e7f7a3790301c6ba5e61\": container with ID starting with e02a4ab64b7d12c2be335320aed7c7677c3f3914d674e7f7a3790301c6ba5e61 not found: ID does not exist" containerID="e02a4ab64b7d12c2be335320aed7c7677c3f3914d674e7f7a3790301c6ba5e61" Oct 30 11:11:44.595992 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.595959 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e02a4ab64b7d12c2be335320aed7c7677c3f3914d674e7f7a3790301c6ba5e61"} err="failed to get container status \"e02a4ab64b7d12c2be335320aed7c7677c3f3914d674e7f7a3790301c6ba5e61\": rpc error: code = NotFound desc = could not find container \"e02a4ab64b7d12c2be335320aed7c7677c3f3914d674e7f7a3790301c6ba5e61\": container with ID starting with e02a4ab64b7d12c2be335320aed7c7677c3f3914d674e7f7a3790301c6ba5e61 not found: ID does not exist" Oct 30 11:11:44.595992 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.595974 4289 scope.go:122] "RemoveContainer" containerID="77ba23d9377c25b188c49d964a594a298d1ff3366744dc2b698cc841933ca93d" Oct 30 11:11:44.596166 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:11:44.596151 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ba23d9377c25b188c49d964a594a298d1ff3366744dc2b698cc841933ca93d\": container with ID starting with 77ba23d9377c25b188c49d964a594a298d1ff3366744dc2b698cc841933ca93d not found: ID does not exist" containerID="77ba23d9377c25b188c49d964a594a298d1ff3366744dc2b698cc841933ca93d" Oct 30 11:11:44.596208 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.596168 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ba23d9377c25b188c49d964a594a298d1ff3366744dc2b698cc841933ca93d"} err="failed to get container status \"77ba23d9377c25b188c49d964a594a298d1ff3366744dc2b698cc841933ca93d\": rpc error: code = NotFound desc = could not find container \"77ba23d9377c25b188c49d964a594a298d1ff3366744dc2b698cc841933ca93d\": container with ID starting with 77ba23d9377c25b188c49d964a594a298d1ff3366744dc2b698cc841933ca93d not found: ID does not exist" Oct 30 11:11:44.613955 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.613937 4289 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75a3f2af-e799-4bfc-bc43-f40ab46c0913-tls-assets\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:44.614049 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.613958 4289 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75a3f2af-e799-4bfc-bc43-f40ab46c0913-config-out\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:44.614049 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.613970 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbx6p\" (UniqueName: \"kubernetes.io/projected/75a3f2af-e799-4bfc-bc43-f40ab46c0913-kube-api-access-dbx6p\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:44.614049 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.613985 4289 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75a3f2af-e799-4bfc-bc43-f40ab46c0913-cluster-tls-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:44.614049 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.613996 4289 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/75a3f2af-e799-4bfc-bc43-f40ab46c0913-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:44.614049 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.614011 4289 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/75a3f2af-e799-4bfc-bc43-f40ab46c0913-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:44.614049 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.614024 4289 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75a3f2af-e799-4bfc-bc43-f40ab46c0913-web-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:11:44.625302 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.625284 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:11:44.629533 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:44.629515 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:11:45.624683 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:45.624649 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a3f2af-e799-4bfc-bc43-f40ab46c0913" path="/var/lib/kubelet/pods/75a3f2af-e799-4bfc-bc43-f40ab46c0913/volumes" Oct 30 11:11:50.556280 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:50.556247 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-m8ln4" Oct 30 11:11:56.257513 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:11:56.257473 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:11:56.468199 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:11:56.468161 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:12:26.257101 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:12:26.257061 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:12:32.163789 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:32.163750 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-m8ln4"] Oct 30 11:12:32.164378 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:32.163958 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-m8ln4" podUID="29a657d4-73ed-479b-88e5-660e99384273" containerName="korrel8r" containerID="cri-o://b3254c7a9593b972aab515bdd08486d6598d65793f784c34393c92a5f1cd5a63" gracePeriod=30 Oct 30 11:12:32.403210 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:32.403187 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-m8ln4" Oct 30 11:12:32.496418 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:32.496388 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29a657d4-73ed-479b-88e5-660e99384273-tls\" (UniqueName: \"kubernetes.io/secret/29a657d4-73ed-479b-88e5-660e99384273-tls\") pod \"29a657d4-73ed-479b-88e5-660e99384273\" (UID: \"29a657d4-73ed-479b-88e5-660e99384273\") " Oct 30 11:12:32.496649 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:32.496460 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a657d4-73ed-479b-88e5-660e99384273-kube-api-access-mpqgr\" (UniqueName: \"kubernetes.io/projected/29a657d4-73ed-479b-88e5-660e99384273-kube-api-access-mpqgr\") pod \"29a657d4-73ed-479b-88e5-660e99384273\" (UID: \"29a657d4-73ed-479b-88e5-660e99384273\") " Oct 30 11:12:32.496649 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:32.496500 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29a657d4-73ed-479b-88e5-660e99384273-config\" (UniqueName: \"kubernetes.io/configmap/29a657d4-73ed-479b-88e5-660e99384273-config\") pod \"29a657d4-73ed-479b-88e5-660e99384273\" (UID: \"29a657d4-73ed-479b-88e5-660e99384273\") " Oct 30 11:12:32.496889 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:32.496868 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a657d4-73ed-479b-88e5-660e99384273-config" pod "29a657d4-73ed-479b-88e5-660e99384273" (UID: "29a657d4-73ed-479b-88e5-660e99384273"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:12:32.498614 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:32.498588 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a657d4-73ed-479b-88e5-660e99384273-kube-api-access-mpqgr" pod "29a657d4-73ed-479b-88e5-660e99384273" (UID: "29a657d4-73ed-479b-88e5-660e99384273"). InnerVolumeSpecName "kube-api-access-mpqgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:12:32.498723 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:32.498624 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a657d4-73ed-479b-88e5-660e99384273-tls" pod "29a657d4-73ed-479b-88e5-660e99384273" (UID: "29a657d4-73ed-479b-88e5-660e99384273"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:12:32.597268 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:32.597236 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/29a657d4-73ed-479b-88e5-660e99384273-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:12:32.597268 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:32.597265 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpqgr\" (UniqueName: \"kubernetes.io/projected/29a657d4-73ed-479b-88e5-660e99384273-kube-api-access-mpqgr\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:12:32.597268 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:32.597275 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/29a657d4-73ed-479b-88e5-660e99384273-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:12:32.660605 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:32.660570 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:12:32.767130 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:32.767049 4289 generic.go:356] "Generic (PLEG): container finished" podID="29a657d4-73ed-479b-88e5-660e99384273" containerID="b3254c7a9593b972aab515bdd08486d6598d65793f784c34393c92a5f1cd5a63" exitCode=0 Oct 30 11:12:32.767130 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:32.767097 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-m8ln4" Oct 30 11:12:32.767306 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:32.767137 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-m8ln4" event={"ID":"29a657d4-73ed-479b-88e5-660e99384273","Type":"ContainerDied","Data":"b3254c7a9593b972aab515bdd08486d6598d65793f784c34393c92a5f1cd5a63"} Oct 30 11:12:32.767306 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:32.767189 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-m8ln4" event={"ID":"29a657d4-73ed-479b-88e5-660e99384273","Type":"ContainerDied","Data":"4defc040a5f748b8e2b085ec52655d8179e83ec37a1b3498f8dd1e17b3d122cc"} Oct 30 11:12:32.767306 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:32.767216 4289 scope.go:122] "RemoveContainer" containerID="b3254c7a9593b972aab515bdd08486d6598d65793f784c34393c92a5f1cd5a63" Oct 30 11:12:32.775414 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:32.775396 4289 scope.go:122] "RemoveContainer" containerID="b3254c7a9593b972aab515bdd08486d6598d65793f784c34393c92a5f1cd5a63" Oct 30 11:12:32.775707 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:12:32.775687 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3254c7a9593b972aab515bdd08486d6598d65793f784c34393c92a5f1cd5a63\": container with ID starting with b3254c7a9593b972aab515bdd08486d6598d65793f784c34393c92a5f1cd5a63 not found: ID does not exist" containerID="b3254c7a9593b972aab515bdd08486d6598d65793f784c34393c92a5f1cd5a63" Oct 30 11:12:32.775760 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:32.775716 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3254c7a9593b972aab515bdd08486d6598d65793f784c34393c92a5f1cd5a63"} err="failed to get container status \"b3254c7a9593b972aab515bdd08486d6598d65793f784c34393c92a5f1cd5a63\": rpc error: code = NotFound desc = could not find container \"b3254c7a9593b972aab515bdd08486d6598d65793f784c34393c92a5f1cd5a63\": container with ID starting with b3254c7a9593b972aab515bdd08486d6598d65793f784c34393c92a5f1cd5a63 not found: ID does not exist" Oct 30 11:12:32.817764 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:32.817731 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-m8ln4"] Oct 30 11:12:32.823558 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:32.823531 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-m8ln4"] Oct 30 11:12:32.831930 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:32.831901 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:12:33.107774 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:33.107693 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:12:33.273452 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:33.273386 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:12:33.624113 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:33.624078 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29a657d4-73ed-479b-88e5-660e99384273" path="/var/lib/kubelet/pods/29a657d4-73ed-479b-88e5-660e99384273/volumes" Oct 30 11:12:35.124538 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.124505 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-sp9mf"] Oct 30 11:12:35.124953 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.124820 4289 cpu_manager.go:414] "containerMap: removing container" podUID="75a3f2af-e799-4bfc-bc43-f40ab46c0913" containerName="alertmanager" Oct 30 11:12:35.124953 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.124832 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a3f2af-e799-4bfc-bc43-f40ab46c0913" containerName="alertmanager" Oct 30 11:12:35.124953 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.124843 4289 cpu_manager.go:414] "containerMap: removing container" podUID="29a657d4-73ed-479b-88e5-660e99384273" containerName="korrel8r" Oct 30 11:12:35.124953 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.124848 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a657d4-73ed-479b-88e5-660e99384273" containerName="korrel8r" Oct 30 11:12:35.124953 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.124854 4289 cpu_manager.go:414] "containerMap: removing container" podUID="75a3f2af-e799-4bfc-bc43-f40ab46c0913" containerName="config-reloader" Oct 30 11:12:35.124953 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.124860 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a3f2af-e799-4bfc-bc43-f40ab46c0913" containerName="config-reloader" Oct 30 11:12:35.124953 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.124870 4289 cpu_manager.go:414] "containerMap: removing container" podUID="75a3f2af-e799-4bfc-bc43-f40ab46c0913" containerName="init-config-reloader" Oct 30 11:12:35.124953 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.124874 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a3f2af-e799-4bfc-bc43-f40ab46c0913" containerName="init-config-reloader" Oct 30 11:12:35.124953 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.124931 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a3f2af-e799-4bfc-bc43-f40ab46c0913" containerName="alertmanager" Oct 30 11:12:35.124953 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.124938 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a657d4-73ed-479b-88e5-660e99384273" containerName="korrel8r" Oct 30 11:12:35.124953 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.124943 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a3f2af-e799-4bfc-bc43-f40ab46c0913" containerName="config-reloader" Oct 30 11:12:35.126079 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.126052 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:12:35.129257 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.129237 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-sp9mf" Oct 30 11:12:35.156158 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.156132 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 11:12:35.156309 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.156201 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 11:12:35.156309 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.156215 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 11:12:35.156414 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.156343 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-shxcd\"" Oct 30 11:12:35.156414 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.156366 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 11:12:35.157721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.157700 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-sp9mf"] Oct 30 11:12:35.220627 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.220587 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2c7f4dc-d605-4199-9219-e5dc932de0d8-config\") pod \"korrel8r-6644769df5-sp9mf\" (UID: \"b2c7f4dc-d605-4199-9219-e5dc932de0d8\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-sp9mf" Oct 30 11:12:35.220834 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.220752 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b2c7f4dc-d605-4199-9219-e5dc932de0d8-tls\") pod \"korrel8r-6644769df5-sp9mf\" (UID: \"b2c7f4dc-d605-4199-9219-e5dc932de0d8\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-sp9mf" Oct 30 11:12:35.220834 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.220786 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzbgb\" (UniqueName: \"kubernetes.io/projected/b2c7f4dc-d605-4199-9219-e5dc932de0d8-kube-api-access-nzbgb\") pod \"korrel8r-6644769df5-sp9mf\" (UID: \"b2c7f4dc-d605-4199-9219-e5dc932de0d8\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-sp9mf" Oct 30 11:12:35.286483 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.286420 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:12:35.321636 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.321609 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2c7f4dc-d605-4199-9219-e5dc932de0d8-config\") pod \"korrel8r-6644769df5-sp9mf\" (UID: \"b2c7f4dc-d605-4199-9219-e5dc932de0d8\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-sp9mf" Oct 30 11:12:35.321772 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.321699 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b2c7f4dc-d605-4199-9219-e5dc932de0d8-tls\") pod \"korrel8r-6644769df5-sp9mf\" (UID: \"b2c7f4dc-d605-4199-9219-e5dc932de0d8\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-sp9mf" Oct 30 11:12:35.321772 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.321723 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzbgb\" (UniqueName: \"kubernetes.io/projected/b2c7f4dc-d605-4199-9219-e5dc932de0d8-kube-api-access-nzbgb\") pod \"korrel8r-6644769df5-sp9mf\" (UID: \"b2c7f4dc-d605-4199-9219-e5dc932de0d8\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-sp9mf" Oct 30 11:12:35.322370 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.322344 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2c7f4dc-d605-4199-9219-e5dc932de0d8-config\") pod \"korrel8r-6644769df5-sp9mf\" (UID: \"b2c7f4dc-d605-4199-9219-e5dc932de0d8\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-sp9mf" Oct 30 11:12:35.324043 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.324018 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b2c7f4dc-d605-4199-9219-e5dc932de0d8-tls\") pod \"korrel8r-6644769df5-sp9mf\" (UID: \"b2c7f4dc-d605-4199-9219-e5dc932de0d8\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-sp9mf" Oct 30 11:12:35.345626 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.345601 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzbgb\" (UniqueName: \"kubernetes.io/projected/b2c7f4dc-d605-4199-9219-e5dc932de0d8-kube-api-access-nzbgb\") pod \"korrel8r-6644769df5-sp9mf\" (UID: \"b2c7f4dc-d605-4199-9219-e5dc932de0d8\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-sp9mf" Oct 30 11:12:35.441626 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.441552 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-sp9mf" Oct 30 11:12:35.456635 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.456603 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:12:35.740234 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.740202 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-sp9mf"] Oct 30 11:12:35.741293 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:12:35.741260 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c7f4dc_d605_4199_9219_e5dc932de0d8.slice/crio-e6b74b6f9d68d2ab237dce1160b29d8a4773e4a220d564cdcf56b4db27469ca2 WatchSource:0}: Error finding container e6b74b6f9d68d2ab237dce1160b29d8a4773e4a220d564cdcf56b4db27469ca2: Status 404 returned error can't find the container with id e6b74b6f9d68d2ab237dce1160b29d8a4773e4a220d564cdcf56b4db27469ca2 Oct 30 11:12:35.782706 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.782678 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-sp9mf" event={"ID":"b2c7f4dc-d605-4199-9219-e5dc932de0d8","Type":"ContainerStarted","Data":"e6b74b6f9d68d2ab237dce1160b29d8a4773e4a220d564cdcf56b4db27469ca2"} Oct 30 11:12:35.790207 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.790184 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:12:35.794526 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.794510 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:12:35.821238 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.821211 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:12:35.838048 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.838023 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rn48g\"" Oct 30 11:12:35.838181 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.838060 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 11:12:35.858041 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.858010 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:12:35.888209 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.888177 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-61f7caf9\"" Oct 30 11:12:35.927299 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.927268 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/73593c0d-be70-4cf1-9f41-647c824f785f-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"73593c0d-be70-4cf1-9f41-647c824f785f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:12:35.927513 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.927317 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/73593c0d-be70-4cf1-9f41-647c824f785f-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"73593c0d-be70-4cf1-9f41-647c824f785f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:12:35.927513 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:35.927467 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78bqf\" (UniqueName: \"kubernetes.io/projected/73593c0d-be70-4cf1-9f41-647c824f785f-kube-api-access-78bqf\") pod \"data-science-collector-collector-1\" (UID: \"73593c0d-be70-4cf1-9f41-647c824f785f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:12:36.025556 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:36.025467 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:12:36.027939 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:36.027915 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/73593c0d-be70-4cf1-9f41-647c824f785f-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"73593c0d-be70-4cf1-9f41-647c824f785f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:12:36.028057 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:36.027966 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/73593c0d-be70-4cf1-9f41-647c824f785f-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"73593c0d-be70-4cf1-9f41-647c824f785f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:12:36.028057 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:36.028016 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78bqf\" (UniqueName: \"kubernetes.io/projected/73593c0d-be70-4cf1-9f41-647c824f785f-kube-api-access-78bqf\") pod \"data-science-collector-collector-1\" (UID: \"73593c0d-be70-4cf1-9f41-647c824f785f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:12:36.028535 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:36.028515 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/73593c0d-be70-4cf1-9f41-647c824f785f-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"73593c0d-be70-4cf1-9f41-647c824f785f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:12:36.030356 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:36.030329 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/73593c0d-be70-4cf1-9f41-647c824f785f-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"73593c0d-be70-4cf1-9f41-647c824f785f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:12:36.050669 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:36.050641 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78bqf\" (UniqueName: \"kubernetes.io/projected/73593c0d-be70-4cf1-9f41-647c824f785f-kube-api-access-78bqf\") pod \"data-science-collector-collector-1\" (UID: \"73593c0d-be70-4cf1-9f41-647c824f785f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:12:36.111515 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:36.111486 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:12:36.402684 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:36.402655 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:12:36.402996 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:12:36.402746 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73593c0d_be70_4cf1_9f41_647c824f785f.slice/crio-44ffe2d950798f74f2cb27909912091a61363592ae4f38ad83f837934a7e69ef WatchSource:0}: Error finding container 44ffe2d950798f74f2cb27909912091a61363592ae4f38ad83f837934a7e69ef: Status 404 returned error can't find the container with id 44ffe2d950798f74f2cb27909912091a61363592ae4f38ad83f837934a7e69ef Oct 30 11:12:36.405054 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:36.405013 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:12:36.405129 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:36.405091 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:12:36.725731 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:36.725679 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:12:36.789042 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:36.789003 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-sp9mf" event={"ID":"b2c7f4dc-d605-4199-9219-e5dc932de0d8","Type":"ContainerStarted","Data":"688e9a22bcf75a041242ccbc317ec9271d4d4e56ed180cdc8da682e8387010eb"} Oct 30 11:12:36.789223 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:36.789126 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-sp9mf" Oct 30 11:12:36.790789 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:36.790765 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"73593c0d-be70-4cf1-9f41-647c824f785f","Type":"ContainerStarted","Data":"c3debea4121b530abb83ae03c3c2ca1c2839cc3c1bd35cd9ec6db1ae0bbc192e"} Oct 30 11:12:36.790910 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:36.790800 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"73593c0d-be70-4cf1-9f41-647c824f785f","Type":"ContainerStarted","Data":"44ffe2d950798f74f2cb27909912091a61363592ae4f38ad83f837934a7e69ef"} Oct 30 11:12:36.829514 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:36.829467 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-sp9mf" podStartSLOduration=1.8294543399999998 podStartE2EDuration="1.82945434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:12:36.828363909 +0000 UTC m=+1647.329743413" watchObservedRunningTime="2026-10-30 11:12:36.82945434 +0000 UTC m=+1647.330833821" Oct 30 11:12:36.870096 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:36.870049 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.870035882 podStartE2EDuration="1.870035882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:12:36.868895249 +0000 UTC m=+1647.370274734" watchObservedRunningTime="2026-10-30 11:12:36.870035882 +0000 UTC m=+1647.371415367" Oct 30 11:12:36.928617 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:36.928581 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:12:47.798529 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:47.798499 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-sp9mf" Oct 30 11:12:50.986344 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:50.986311 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-sp9mf"] Oct 30 11:12:50.986764 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:50.986523 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-sp9mf" podUID="b2c7f4dc-d605-4199-9219-e5dc932de0d8" containerName="korrel8r" containerID="cri-o://688e9a22bcf75a041242ccbc317ec9271d4d4e56ed180cdc8da682e8387010eb" gracePeriod=30 Oct 30 11:12:51.260243 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.260217 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-sp9mf" Oct 30 11:12:51.357177 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.357145 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c7f4dc-d605-4199-9219-e5dc932de0d8-kube-api-access-nzbgb\" (UniqueName: \"kubernetes.io/projected/b2c7f4dc-d605-4199-9219-e5dc932de0d8-kube-api-access-nzbgb\") pod \"b2c7f4dc-d605-4199-9219-e5dc932de0d8\" (UID: \"b2c7f4dc-d605-4199-9219-e5dc932de0d8\") " Oct 30 11:12:51.357365 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.357208 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2c7f4dc-d605-4199-9219-e5dc932de0d8-tls\" (UniqueName: \"kubernetes.io/secret/b2c7f4dc-d605-4199-9219-e5dc932de0d8-tls\") pod \"b2c7f4dc-d605-4199-9219-e5dc932de0d8\" (UID: \"b2c7f4dc-d605-4199-9219-e5dc932de0d8\") " Oct 30 11:12:51.357365 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.357303 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2c7f4dc-d605-4199-9219-e5dc932de0d8-config\" (UniqueName: \"kubernetes.io/configmap/b2c7f4dc-d605-4199-9219-e5dc932de0d8-config\") pod \"b2c7f4dc-d605-4199-9219-e5dc932de0d8\" (UID: \"b2c7f4dc-d605-4199-9219-e5dc932de0d8\") " Oct 30 11:12:51.357884 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.357794 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c7f4dc-d605-4199-9219-e5dc932de0d8-config" pod "b2c7f4dc-d605-4199-9219-e5dc932de0d8" (UID: "b2c7f4dc-d605-4199-9219-e5dc932de0d8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:12:51.360053 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.360027 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2c7f4dc-d605-4199-9219-e5dc932de0d8-tls" pod "b2c7f4dc-d605-4199-9219-e5dc932de0d8" (UID: "b2c7f4dc-d605-4199-9219-e5dc932de0d8"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:12:51.362060 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.362033 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c7f4dc-d605-4199-9219-e5dc932de0d8-kube-api-access-nzbgb" pod "b2c7f4dc-d605-4199-9219-e5dc932de0d8" (UID: "b2c7f4dc-d605-4199-9219-e5dc932de0d8"). InnerVolumeSpecName "kube-api-access-nzbgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:12:51.394609 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.394573 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:12:51.394870 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.394819 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="73593c0d-be70-4cf1-9f41-647c824f785f" containerName="otc-container" containerID="cri-o://c3debea4121b530abb83ae03c3c2ca1c2839cc3c1bd35cd9ec6db1ae0bbc192e" gracePeriod=30 Oct 30 11:12:51.458845 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.458810 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzbgb\" (UniqueName: \"kubernetes.io/projected/b2c7f4dc-d605-4199-9219-e5dc932de0d8-kube-api-access-nzbgb\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:12:51.458845 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.458847 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b2c7f4dc-d605-4199-9219-e5dc932de0d8-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:12:51.459036 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.458861 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2c7f4dc-d605-4199-9219-e5dc932de0d8-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:12:51.517609 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.517577 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:12:51.639304 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.639281 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:12:51.687886 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.687804 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:12:51.761058 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.761030 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73593c0d-be70-4cf1-9f41-647c824f785f-otc-internal\" (UniqueName: \"kubernetes.io/configmap/73593c0d-be70-4cf1-9f41-647c824f785f-otc-internal\") pod \"73593c0d-be70-4cf1-9f41-647c824f785f\" (UID: \"73593c0d-be70-4cf1-9f41-647c824f785f\") " Oct 30 11:12:51.761230 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.761109 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73593c0d-be70-4cf1-9f41-647c824f785f-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/73593c0d-be70-4cf1-9f41-647c824f785f-monitoring-tls-certs\") pod \"73593c0d-be70-4cf1-9f41-647c824f785f\" (UID: \"73593c0d-be70-4cf1-9f41-647c824f785f\") " Oct 30 11:12:51.761230 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.761175 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73593c0d-be70-4cf1-9f41-647c824f785f-kube-api-access-78bqf\" (UniqueName: \"kubernetes.io/projected/73593c0d-be70-4cf1-9f41-647c824f785f-kube-api-access-78bqf\") pod \"73593c0d-be70-4cf1-9f41-647c824f785f\" (UID: \"73593c0d-be70-4cf1-9f41-647c824f785f\") " Oct 30 11:12:51.761424 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.761403 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73593c0d-be70-4cf1-9f41-647c824f785f-otc-internal" pod "73593c0d-be70-4cf1-9f41-647c824f785f" (UID: "73593c0d-be70-4cf1-9f41-647c824f785f"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:12:51.763056 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.763033 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73593c0d-be70-4cf1-9f41-647c824f785f-monitoring-tls-certs" pod "73593c0d-be70-4cf1-9f41-647c824f785f" (UID: "73593c0d-be70-4cf1-9f41-647c824f785f"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:12:51.763145 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.763116 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73593c0d-be70-4cf1-9f41-647c824f785f-kube-api-access-78bqf" pod "73593c0d-be70-4cf1-9f41-647c824f785f" (UID: "73593c0d-be70-4cf1-9f41-647c824f785f"). InnerVolumeSpecName "kube-api-access-78bqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:12:51.854887 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.854800 4289 generic.go:356] "Generic (PLEG): container finished" podID="b2c7f4dc-d605-4199-9219-e5dc932de0d8" containerID="688e9a22bcf75a041242ccbc317ec9271d4d4e56ed180cdc8da682e8387010eb" exitCode=0 Oct 30 11:12:51.854887 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.854839 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-sp9mf" event={"ID":"b2c7f4dc-d605-4199-9219-e5dc932de0d8","Type":"ContainerDied","Data":"688e9a22bcf75a041242ccbc317ec9271d4d4e56ed180cdc8da682e8387010eb"} Oct 30 11:12:51.855104 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.854892 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-sp9mf" event={"ID":"b2c7f4dc-d605-4199-9219-e5dc932de0d8","Type":"ContainerDied","Data":"e6b74b6f9d68d2ab237dce1160b29d8a4773e4a220d564cdcf56b4db27469ca2"} Oct 30 11:12:51.855104 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.854848 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-sp9mf" Oct 30 11:12:51.855104 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.854909 4289 scope.go:122] "RemoveContainer" containerID="688e9a22bcf75a041242ccbc317ec9271d4d4e56ed180cdc8da682e8387010eb" Oct 30 11:12:51.856376 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.856348 4289 generic.go:356] "Generic (PLEG): container finished" podID="73593c0d-be70-4cf1-9f41-647c824f785f" containerID="c3debea4121b530abb83ae03c3c2ca1c2839cc3c1bd35cd9ec6db1ae0bbc192e" exitCode=0 Oct 30 11:12:51.856519 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.856392 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:12:51.856519 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.856409 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"73593c0d-be70-4cf1-9f41-647c824f785f","Type":"ContainerDied","Data":"c3debea4121b530abb83ae03c3c2ca1c2839cc3c1bd35cd9ec6db1ae0bbc192e"} Oct 30 11:12:51.856519 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.856475 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"73593c0d-be70-4cf1-9f41-647c824f785f","Type":"ContainerDied","Data":"44ffe2d950798f74f2cb27909912091a61363592ae4f38ad83f837934a7e69ef"} Oct 30 11:12:51.862215 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.862189 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/73593c0d-be70-4cf1-9f41-647c824f785f-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:12:51.862344 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.862217 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/73593c0d-be70-4cf1-9f41-647c824f785f-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:12:51.862344 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.862234 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78bqf\" (UniqueName: \"kubernetes.io/projected/73593c0d-be70-4cf1-9f41-647c824f785f-kube-api-access-78bqf\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:12:51.864035 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.864020 4289 scope.go:122] "RemoveContainer" containerID="688e9a22bcf75a041242ccbc317ec9271d4d4e56ed180cdc8da682e8387010eb" Oct 30 11:12:51.864340 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:12:51.864306 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"688e9a22bcf75a041242ccbc317ec9271d4d4e56ed180cdc8da682e8387010eb\": container with ID starting with 688e9a22bcf75a041242ccbc317ec9271d4d4e56ed180cdc8da682e8387010eb not found: ID does not exist" containerID="688e9a22bcf75a041242ccbc317ec9271d4d4e56ed180cdc8da682e8387010eb" Oct 30 11:12:51.864402 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.864353 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"688e9a22bcf75a041242ccbc317ec9271d4d4e56ed180cdc8da682e8387010eb"} err="failed to get container status \"688e9a22bcf75a041242ccbc317ec9271d4d4e56ed180cdc8da682e8387010eb\": rpc error: code = NotFound desc = could not find container \"688e9a22bcf75a041242ccbc317ec9271d4d4e56ed180cdc8da682e8387010eb\": container with ID starting with 688e9a22bcf75a041242ccbc317ec9271d4d4e56ed180cdc8da682e8387010eb not found: ID does not exist" Oct 30 11:12:51.864402 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.864378 4289 scope.go:122] "RemoveContainer" containerID="c3debea4121b530abb83ae03c3c2ca1c2839cc3c1bd35cd9ec6db1ae0bbc192e" Oct 30 11:12:51.872242 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.872225 4289 scope.go:122] "RemoveContainer" containerID="c3debea4121b530abb83ae03c3c2ca1c2839cc3c1bd35cd9ec6db1ae0bbc192e" Oct 30 11:12:51.872544 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:12:51.872500 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3debea4121b530abb83ae03c3c2ca1c2839cc3c1bd35cd9ec6db1ae0bbc192e\": container with ID starting with c3debea4121b530abb83ae03c3c2ca1c2839cc3c1bd35cd9ec6db1ae0bbc192e not found: ID does not exist" containerID="c3debea4121b530abb83ae03c3c2ca1c2839cc3c1bd35cd9ec6db1ae0bbc192e" Oct 30 11:12:51.872544 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.872527 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3debea4121b530abb83ae03c3c2ca1c2839cc3c1bd35cd9ec6db1ae0bbc192e"} err="failed to get container status \"c3debea4121b530abb83ae03c3c2ca1c2839cc3c1bd35cd9ec6db1ae0bbc192e\": rpc error: code = NotFound desc = could not find container \"c3debea4121b530abb83ae03c3c2ca1c2839cc3c1bd35cd9ec6db1ae0bbc192e\": container with ID starting with c3debea4121b530abb83ae03c3c2ca1c2839cc3c1bd35cd9ec6db1ae0bbc192e not found: ID does not exist" Oct 30 11:12:51.904549 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.904521 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:12:51.912111 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.912085 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:12:51.935865 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.935831 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:12:51.945642 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.945618 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-sp9mf"] Oct 30 11:12:51.949780 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:51.949756 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-sp9mf"] Oct 30 11:12:52.096247 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:52.096219 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:12:53.623804 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:53.623733 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73593c0d-be70-4cf1-9f41-647c824f785f" path="/var/lib/kubelet/pods/73593c0d-be70-4cf1-9f41-647c824f785f/volumes" Oct 30 11:12:53.624182 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:53.624107 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2c7f4dc-d605-4199-9219-e5dc932de0d8" path="/var/lib/kubelet/pods/b2c7f4dc-d605-4199-9219-e5dc932de0d8/volumes" Oct 30 11:12:53.938140 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:53.938048 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:12:53.938998 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:53.938979 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-rjsqj"] Oct 30 11:12:53.939288 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:53.939277 4289 cpu_manager.go:414] "containerMap: removing container" podUID="b2c7f4dc-d605-4199-9219-e5dc932de0d8" containerName="korrel8r" Oct 30 11:12:53.939321 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:53.939290 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c7f4dc-d605-4199-9219-e5dc932de0d8" containerName="korrel8r" Oct 30 11:12:53.939321 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:53.939304 4289 cpu_manager.go:414] "containerMap: removing container" podUID="73593c0d-be70-4cf1-9f41-647c824f785f" containerName="otc-container" Oct 30 11:12:53.939321 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:53.939310 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73593c0d-be70-4cf1-9f41-647c824f785f" containerName="otc-container" Oct 30 11:12:53.939410 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:53.939370 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c7f4dc-d605-4199-9219-e5dc932de0d8" containerName="korrel8r" Oct 30 11:12:53.939410 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:53.939380 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="73593c0d-be70-4cf1-9f41-647c824f785f" containerName="otc-container" Oct 30 11:12:53.943591 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:53.943577 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-rjsqj" Oct 30 11:12:53.961063 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:53.961029 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 11:12:53.970553 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:53.970522 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 11:12:53.970690 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:53.970537 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-jbt4h\"" Oct 30 11:12:53.973209 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:53.973177 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-rjsqj"] Oct 30 11:12:53.976670 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:53.976649 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 11:12:53.976787 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:53.976736 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 11:12:54.081484 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.081448 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx5jx\" (UniqueName: \"kubernetes.io/projected/47e3a8ed-61ee-4e76-bf25-8bf2240d0664-kube-api-access-mx5jx\") pod \"korrel8r-6644769df5-rjsqj\" (UID: \"47e3a8ed-61ee-4e76-bf25-8bf2240d0664\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-rjsqj" Oct 30 11:12:54.081646 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.081576 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47e3a8ed-61ee-4e76-bf25-8bf2240d0664-config\") pod \"korrel8r-6644769df5-rjsqj\" (UID: \"47e3a8ed-61ee-4e76-bf25-8bf2240d0664\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-rjsqj" Oct 30 11:12:54.081646 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.081633 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/47e3a8ed-61ee-4e76-bf25-8bf2240d0664-tls\") pod \"korrel8r-6644769df5-rjsqj\" (UID: \"47e3a8ed-61ee-4e76-bf25-8bf2240d0664\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-rjsqj" Oct 30 11:12:54.113814 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.113784 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:12:54.182361 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.182331 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47e3a8ed-61ee-4e76-bf25-8bf2240d0664-config\") pod \"korrel8r-6644769df5-rjsqj\" (UID: \"47e3a8ed-61ee-4e76-bf25-8bf2240d0664\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-rjsqj" Oct 30 11:12:54.182361 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.182366 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/47e3a8ed-61ee-4e76-bf25-8bf2240d0664-tls\") pod \"korrel8r-6644769df5-rjsqj\" (UID: \"47e3a8ed-61ee-4e76-bf25-8bf2240d0664\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-rjsqj" Oct 30 11:12:54.182599 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.182396 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx5jx\" (UniqueName: \"kubernetes.io/projected/47e3a8ed-61ee-4e76-bf25-8bf2240d0664-kube-api-access-mx5jx\") pod \"korrel8r-6644769df5-rjsqj\" (UID: \"47e3a8ed-61ee-4e76-bf25-8bf2240d0664\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-rjsqj" Oct 30 11:12:54.183074 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.183049 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47e3a8ed-61ee-4e76-bf25-8bf2240d0664-config\") pod \"korrel8r-6644769df5-rjsqj\" (UID: \"47e3a8ed-61ee-4e76-bf25-8bf2240d0664\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-rjsqj" Oct 30 11:12:54.184718 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.184693 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/47e3a8ed-61ee-4e76-bf25-8bf2240d0664-tls\") pod \"korrel8r-6644769df5-rjsqj\" (UID: \"47e3a8ed-61ee-4e76-bf25-8bf2240d0664\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-rjsqj" Oct 30 11:12:54.205346 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.205290 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx5jx\" (UniqueName: \"kubernetes.io/projected/47e3a8ed-61ee-4e76-bf25-8bf2240d0664-kube-api-access-mx5jx\") pod \"korrel8r-6644769df5-rjsqj\" (UID: \"47e3a8ed-61ee-4e76-bf25-8bf2240d0664\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-rjsqj" Oct 30 11:12:54.256200 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.256173 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-rjsqj" Oct 30 11:12:54.550024 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.549992 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-rjsqj"] Oct 30 11:12:54.551163 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:12:54.551126 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e3a8ed_61ee_4e76_bf25_8bf2240d0664.slice/crio-56f21c51260192bd85d55e49372e63a7f288b495b65fd50c35c3405cf971d91c WatchSource:0}: Error finding container 56f21c51260192bd85d55e49372e63a7f288b495b65fd50c35c3405cf971d91c: Status 404 returned error can't find the container with id 56f21c51260192bd85d55e49372e63a7f288b495b65fd50c35c3405cf971d91c Oct 30 11:12:54.593235 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.593205 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:12:54.597812 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.597794 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:12:54.617314 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.617290 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:12:54.640313 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.640284 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:12:54.686332 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.686305 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d963ef2-9501-4560-a7cf-24a7aaa17ab2-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6d963ef2-9501-4560-a7cf-24a7aaa17ab2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:12:54.686488 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.686361 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6d963ef2-9501-4560-a7cf-24a7aaa17ab2-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"6d963ef2-9501-4560-a7cf-24a7aaa17ab2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:12:54.686488 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.686451 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6bmz\" (UniqueName: \"kubernetes.io/projected/6d963ef2-9501-4560-a7cf-24a7aaa17ab2-kube-api-access-k6bmz\") pod \"data-science-collector-collector-1\" (UID: \"6d963ef2-9501-4560-a7cf-24a7aaa17ab2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:12:54.689160 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.689142 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-61f7caf9\"" Oct 30 11:12:54.689253 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.689236 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 11:12:54.689327 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.689313 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-vdrlt\"" Oct 30 11:12:54.787532 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.787494 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d963ef2-9501-4560-a7cf-24a7aaa17ab2-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6d963ef2-9501-4560-a7cf-24a7aaa17ab2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:12:54.787705 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.787595 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6d963ef2-9501-4560-a7cf-24a7aaa17ab2-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"6d963ef2-9501-4560-a7cf-24a7aaa17ab2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:12:54.787705 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.787626 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6bmz\" (UniqueName: \"kubernetes.io/projected/6d963ef2-9501-4560-a7cf-24a7aaa17ab2-kube-api-access-k6bmz\") pod \"data-science-collector-collector-1\" (UID: \"6d963ef2-9501-4560-a7cf-24a7aaa17ab2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:12:54.788479 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.788455 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6d963ef2-9501-4560-a7cf-24a7aaa17ab2-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"6d963ef2-9501-4560-a7cf-24a7aaa17ab2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:12:54.789912 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.789895 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d963ef2-9501-4560-a7cf-24a7aaa17ab2-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6d963ef2-9501-4560-a7cf-24a7aaa17ab2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:12:54.810388 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.810322 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6bmz\" (UniqueName: \"kubernetes.io/projected/6d963ef2-9501-4560-a7cf-24a7aaa17ab2-kube-api-access-k6bmz\") pod \"data-science-collector-collector-1\" (UID: \"6d963ef2-9501-4560-a7cf-24a7aaa17ab2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:12:54.818445 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.818408 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:12:54.872654 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.872626 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-rjsqj" event={"ID":"47e3a8ed-61ee-4e76-bf25-8bf2240d0664","Type":"ContainerStarted","Data":"65f5c2f4d3c9f89f283b0fdd6d7a9e09a0688757d98dd2aa79d8b130f787b2eb"} Oct 30 11:12:54.872759 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.872674 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-rjsqj" event={"ID":"47e3a8ed-61ee-4e76-bf25-8bf2240d0664","Type":"ContainerStarted","Data":"56f21c51260192bd85d55e49372e63a7f288b495b65fd50c35c3405cf971d91c"} Oct 30 11:12:54.872797 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.872769 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-rjsqj" Oct 30 11:12:54.914112 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.914069 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-rjsqj" podStartSLOduration=1.914057079 podStartE2EDuration="1.914057079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:12:54.913367638 +0000 UTC m=+1665.414747129" watchObservedRunningTime="2026-10-30 11:12:54.914057079 +0000 UTC m=+1665.415436554" Oct 30 11:12:54.926451 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:54.926410 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:12:55.218048 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:55.218023 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:12:55.218420 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:12:55.218396 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d963ef2_9501_4560_a7cf_24a7aaa17ab2.slice/crio-588f7cc5840027dcb7a208e52cd6e33e4e21104899a4aad86b5f9ab88d7b19b1 WatchSource:0}: Error finding container 588f7cc5840027dcb7a208e52cd6e33e4e21104899a4aad86b5f9ab88d7b19b1: Status 404 returned error can't find the container with id 588f7cc5840027dcb7a208e52cd6e33e4e21104899a4aad86b5f9ab88d7b19b1 Oct 30 11:12:55.221038 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:55.221006 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:12:55.221097 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:55.221063 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:12:55.754552 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:55.754516 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:12:55.879250 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:55.879214 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6d963ef2-9501-4560-a7cf-24a7aaa17ab2","Type":"ContainerStarted","Data":"fbf8166e2bd446629c6e9e0c00ca4b3edde3292ac4b07b24b3f84fa1f7cbe17a"} Oct 30 11:12:55.879455 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:55.879276 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6d963ef2-9501-4560-a7cf-24a7aaa17ab2","Type":"ContainerStarted","Data":"588f7cc5840027dcb7a208e52cd6e33e4e21104899a4aad86b5f9ab88d7b19b1"} Oct 30 11:12:55.920412 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:55.920366 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.92035211 podStartE2EDuration="1.92035211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:12:55.918725859 +0000 UTC m=+1666.420105343" watchObservedRunningTime="2026-10-30 11:12:55.92035211 +0000 UTC m=+1666.421731594" Oct 30 11:12:55.935746 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:55.935711 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:12:56.257458 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:12:56.257405 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:12:56.576890 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:12:56.576794 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:13:05.881469 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:05.881416 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-rjsqj" Oct 30 11:13:26.257651 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:13:26.257616 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:13:30.034309 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:30.034269 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:13:50.182229 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:50.182193 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-rjsqj"] Oct 30 11:13:50.182691 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:50.182414 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-rjsqj" podUID="47e3a8ed-61ee-4e76-bf25-8bf2240d0664" containerName="korrel8r" containerID="cri-o://65f5c2f4d3c9f89f283b0fdd6d7a9e09a0688757d98dd2aa79d8b130f787b2eb" gracePeriod=30 Oct 30 11:13:50.428910 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:50.428884 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-rjsqj" Oct 30 11:13:50.547533 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:50.547503 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47e3a8ed-61ee-4e76-bf25-8bf2240d0664-tls\" (UniqueName: \"kubernetes.io/secret/47e3a8ed-61ee-4e76-bf25-8bf2240d0664-tls\") pod \"47e3a8ed-61ee-4e76-bf25-8bf2240d0664\" (UID: \"47e3a8ed-61ee-4e76-bf25-8bf2240d0664\") " Oct 30 11:13:50.547724 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:50.547576 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e3a8ed-61ee-4e76-bf25-8bf2240d0664-config\" (UniqueName: \"kubernetes.io/configmap/47e3a8ed-61ee-4e76-bf25-8bf2240d0664-config\") pod \"47e3a8ed-61ee-4e76-bf25-8bf2240d0664\" (UID: \"47e3a8ed-61ee-4e76-bf25-8bf2240d0664\") " Oct 30 11:13:50.547724 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:50.547689 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e3a8ed-61ee-4e76-bf25-8bf2240d0664-kube-api-access-mx5jx\" (UniqueName: \"kubernetes.io/projected/47e3a8ed-61ee-4e76-bf25-8bf2240d0664-kube-api-access-mx5jx\") pod \"47e3a8ed-61ee-4e76-bf25-8bf2240d0664\" (UID: \"47e3a8ed-61ee-4e76-bf25-8bf2240d0664\") " Oct 30 11:13:50.547998 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:50.547973 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e3a8ed-61ee-4e76-bf25-8bf2240d0664-config" pod "47e3a8ed-61ee-4e76-bf25-8bf2240d0664" (UID: "47e3a8ed-61ee-4e76-bf25-8bf2240d0664"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:13:50.551294 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:50.551266 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e3a8ed-61ee-4e76-bf25-8bf2240d0664-kube-api-access-mx5jx" pod "47e3a8ed-61ee-4e76-bf25-8bf2240d0664" (UID: "47e3a8ed-61ee-4e76-bf25-8bf2240d0664"). InnerVolumeSpecName "kube-api-access-mx5jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:13:50.558219 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:50.558192 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47e3a8ed-61ee-4e76-bf25-8bf2240d0664-tls" pod "47e3a8ed-61ee-4e76-bf25-8bf2240d0664" (UID: "47e3a8ed-61ee-4e76-bf25-8bf2240d0664"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:13:50.578536 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:50.578513 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:13:50.578726 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:50.578704 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="6d963ef2-9501-4560-a7cf-24a7aaa17ab2" containerName="otc-container" containerID="cri-o://fbf8166e2bd446629c6e9e0c00ca4b3edde3292ac4b07b24b3f84fa1f7cbe17a" gracePeriod=30 Oct 30 11:13:50.648872 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:50.648845 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/47e3a8ed-61ee-4e76-bf25-8bf2240d0664-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:13:50.648872 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:50.648868 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47e3a8ed-61ee-4e76-bf25-8bf2240d0664-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:13:50.648872 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:50.648878 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx5jx\" (UniqueName: \"kubernetes.io/projected/47e3a8ed-61ee-4e76-bf25-8bf2240d0664-kube-api-access-mx5jx\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:13:50.701131 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:50.701093 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:13:50.816424 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:50.816400 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:13:50.872349 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:50.872308 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:13:50.950954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:50.950925 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d963ef2-9501-4560-a7cf-24a7aaa17ab2-otc-internal\" (UniqueName: \"kubernetes.io/configmap/6d963ef2-9501-4560-a7cf-24a7aaa17ab2-otc-internal\") pod \"6d963ef2-9501-4560-a7cf-24a7aaa17ab2\" (UID: \"6d963ef2-9501-4560-a7cf-24a7aaa17ab2\") " Oct 30 11:13:50.951118 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:50.950983 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d963ef2-9501-4560-a7cf-24a7aaa17ab2-kube-api-access-k6bmz\" (UniqueName: \"kubernetes.io/projected/6d963ef2-9501-4560-a7cf-24a7aaa17ab2-kube-api-access-k6bmz\") pod \"6d963ef2-9501-4560-a7cf-24a7aaa17ab2\" (UID: \"6d963ef2-9501-4560-a7cf-24a7aaa17ab2\") " Oct 30 11:13:50.951118 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:50.951027 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d963ef2-9501-4560-a7cf-24a7aaa17ab2-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d963ef2-9501-4560-a7cf-24a7aaa17ab2-monitoring-tls-certs\") pod \"6d963ef2-9501-4560-a7cf-24a7aaa17ab2\" (UID: \"6d963ef2-9501-4560-a7cf-24a7aaa17ab2\") " Oct 30 11:13:50.951353 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:50.951326 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d963ef2-9501-4560-a7cf-24a7aaa17ab2-otc-internal" pod "6d963ef2-9501-4560-a7cf-24a7aaa17ab2" (UID: "6d963ef2-9501-4560-a7cf-24a7aaa17ab2"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:13:50.953043 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:50.953017 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d963ef2-9501-4560-a7cf-24a7aaa17ab2-kube-api-access-k6bmz" pod "6d963ef2-9501-4560-a7cf-24a7aaa17ab2" (UID: "6d963ef2-9501-4560-a7cf-24a7aaa17ab2"). InnerVolumeSpecName "kube-api-access-k6bmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:13:50.953153 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:50.953123 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d963ef2-9501-4560-a7cf-24a7aaa17ab2-monitoring-tls-certs" pod "6d963ef2-9501-4560-a7cf-24a7aaa17ab2" (UID: "6d963ef2-9501-4560-a7cf-24a7aaa17ab2"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:13:51.051630 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.051595 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6d963ef2-9501-4560-a7cf-24a7aaa17ab2-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:13:51.051630 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.051631 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6bmz\" (UniqueName: \"kubernetes.io/projected/6d963ef2-9501-4560-a7cf-24a7aaa17ab2-kube-api-access-k6bmz\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:13:51.051901 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.051648 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d963ef2-9501-4560-a7cf-24a7aaa17ab2-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:13:51.104837 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.104752 4289 generic.go:356] "Generic (PLEG): container finished" podID="47e3a8ed-61ee-4e76-bf25-8bf2240d0664" containerID="65f5c2f4d3c9f89f283b0fdd6d7a9e09a0688757d98dd2aa79d8b130f787b2eb" exitCode=0 Oct 30 11:13:51.104837 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.104803 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-rjsqj" Oct 30 11:13:51.105054 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.104834 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-rjsqj" event={"ID":"47e3a8ed-61ee-4e76-bf25-8bf2240d0664","Type":"ContainerDied","Data":"65f5c2f4d3c9f89f283b0fdd6d7a9e09a0688757d98dd2aa79d8b130f787b2eb"} Oct 30 11:13:51.105054 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.104900 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-rjsqj" event={"ID":"47e3a8ed-61ee-4e76-bf25-8bf2240d0664","Type":"ContainerDied","Data":"56f21c51260192bd85d55e49372e63a7f288b495b65fd50c35c3405cf971d91c"} Oct 30 11:13:51.105054 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.104929 4289 scope.go:122] "RemoveContainer" containerID="65f5c2f4d3c9f89f283b0fdd6d7a9e09a0688757d98dd2aa79d8b130f787b2eb" Oct 30 11:13:51.106307 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.106284 4289 generic.go:356] "Generic (PLEG): container finished" podID="6d963ef2-9501-4560-a7cf-24a7aaa17ab2" containerID="fbf8166e2bd446629c6e9e0c00ca4b3edde3292ac4b07b24b3f84fa1f7cbe17a" exitCode=0 Oct 30 11:13:51.106449 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.106328 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:13:51.106449 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.106342 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6d963ef2-9501-4560-a7cf-24a7aaa17ab2","Type":"ContainerDied","Data":"fbf8166e2bd446629c6e9e0c00ca4b3edde3292ac4b07b24b3f84fa1f7cbe17a"} Oct 30 11:13:51.106449 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.106380 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6d963ef2-9501-4560-a7cf-24a7aaa17ab2","Type":"ContainerDied","Data":"588f7cc5840027dcb7a208e52cd6e33e4e21104899a4aad86b5f9ab88d7b19b1"} Oct 30 11:13:51.115214 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.115192 4289 scope.go:122] "RemoveContainer" containerID="65f5c2f4d3c9f89f283b0fdd6d7a9e09a0688757d98dd2aa79d8b130f787b2eb" Oct 30 11:13:51.115628 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:13:51.115593 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f5c2f4d3c9f89f283b0fdd6d7a9e09a0688757d98dd2aa79d8b130f787b2eb\": container with ID starting with 65f5c2f4d3c9f89f283b0fdd6d7a9e09a0688757d98dd2aa79d8b130f787b2eb not found: ID does not exist" containerID="65f5c2f4d3c9f89f283b0fdd6d7a9e09a0688757d98dd2aa79d8b130f787b2eb" Oct 30 11:13:51.115729 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.115642 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f5c2f4d3c9f89f283b0fdd6d7a9e09a0688757d98dd2aa79d8b130f787b2eb"} err="failed to get container status \"65f5c2f4d3c9f89f283b0fdd6d7a9e09a0688757d98dd2aa79d8b130f787b2eb\": rpc error: code = NotFound desc = could not find container \"65f5c2f4d3c9f89f283b0fdd6d7a9e09a0688757d98dd2aa79d8b130f787b2eb\": container with ID starting with 65f5c2f4d3c9f89f283b0fdd6d7a9e09a0688757d98dd2aa79d8b130f787b2eb not found: ID does not exist" Oct 30 11:13:51.115729 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.115667 4289 scope.go:122] "RemoveContainer" containerID="fbf8166e2bd446629c6e9e0c00ca4b3edde3292ac4b07b24b3f84fa1f7cbe17a" Oct 30 11:13:51.125483 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.125464 4289 scope.go:122] "RemoveContainer" containerID="fbf8166e2bd446629c6e9e0c00ca4b3edde3292ac4b07b24b3f84fa1f7cbe17a" Oct 30 11:13:51.125793 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:13:51.125775 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbf8166e2bd446629c6e9e0c00ca4b3edde3292ac4b07b24b3f84fa1f7cbe17a\": container with ID starting with fbf8166e2bd446629c6e9e0c00ca4b3edde3292ac4b07b24b3f84fa1f7cbe17a not found: ID does not exist" containerID="fbf8166e2bd446629c6e9e0c00ca4b3edde3292ac4b07b24b3f84fa1f7cbe17a" Oct 30 11:13:51.125869 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.125800 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbf8166e2bd446629c6e9e0c00ca4b3edde3292ac4b07b24b3f84fa1f7cbe17a"} err="failed to get container status \"fbf8166e2bd446629c6e9e0c00ca4b3edde3292ac4b07b24b3f84fa1f7cbe17a\": rpc error: code = NotFound desc = could not find container \"fbf8166e2bd446629c6e9e0c00ca4b3edde3292ac4b07b24b3f84fa1f7cbe17a\": container with ID starting with fbf8166e2bd446629c6e9e0c00ca4b3edde3292ac4b07b24b3f84fa1f7cbe17a not found: ID does not exist" Oct 30 11:13:51.138865 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.138826 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:13:51.157353 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.157330 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:13:51.161222 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.161202 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:13:51.195113 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.195093 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-rjsqj"] Oct 30 11:13:51.202107 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.202088 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-rjsqj"] Oct 30 11:13:51.317520 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.317491 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:13:51.623608 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.623573 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e3a8ed-61ee-4e76-bf25-8bf2240d0664" path="/var/lib/kubelet/pods/47e3a8ed-61ee-4e76-bf25-8bf2240d0664/volumes" Oct 30 11:13:51.624032 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:51.624013 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d963ef2-9501-4560-a7cf-24a7aaa17ab2" path="/var/lib/kubelet/pods/6d963ef2-9501-4560-a7cf-24a7aaa17ab2/volumes" Oct 30 11:13:54.429653 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:54.429622 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:13:56.257233 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:13:56.257198 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:13:56.700798 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:13:56.700711 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:14:13.738157 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:13.738106 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:14:17.796744 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.796705 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:14:17.806208 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.806179 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m"] Oct 30 11:14:17.806501 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.806489 4289 cpu_manager.go:414] "containerMap: removing container" podUID="6d963ef2-9501-4560-a7cf-24a7aaa17ab2" containerName="otc-container" Oct 30 11:14:17.806556 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.806502 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d963ef2-9501-4560-a7cf-24a7aaa17ab2" containerName="otc-container" Oct 30 11:14:17.806556 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.806531 4289 cpu_manager.go:414] "containerMap: removing container" podUID="47e3a8ed-61ee-4e76-bf25-8bf2240d0664" containerName="korrel8r" Oct 30 11:14:17.806556 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.806536 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e3a8ed-61ee-4e76-bf25-8bf2240d0664" containerName="korrel8r" Oct 30 11:14:17.806647 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.806597 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e3a8ed-61ee-4e76-bf25-8bf2240d0664" containerName="korrel8r" Oct 30 11:14:17.806647 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.806607 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d963ef2-9501-4560-a7cf-24a7aaa17ab2" containerName="otc-container" Oct 30 11:14:17.810758 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.810738 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m" Oct 30 11:14:17.837804 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.837775 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 11:14:17.839105 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.839084 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m"] Oct 30 11:14:17.840157 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.840141 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 11:14:17.840422 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.840409 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 11:14:17.840472 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.840410 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 11:14:17.840472 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.840462 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-zvxpm\"" Oct 30 11:14:17.848882 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.848862 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd4zc\" (UniqueName: \"kubernetes.io/projected/b1ed3274-9c04-4de7-b397-a8d53269aee4-kube-api-access-gd4zc\") pod \"korrel8r-5d45699dc4-qwh8m\" (UID: \"b1ed3274-9c04-4de7-b397-a8d53269aee4\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m" Oct 30 11:14:17.848962 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.848893 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1ed3274-9c04-4de7-b397-a8d53269aee4-config\") pod \"korrel8r-5d45699dc4-qwh8m\" (UID: \"b1ed3274-9c04-4de7-b397-a8d53269aee4\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m" Oct 30 11:14:17.849022 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.849007 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b1ed3274-9c04-4de7-b397-a8d53269aee4-tls\") pod \"korrel8r-5d45699dc4-qwh8m\" (UID: \"b1ed3274-9c04-4de7-b397-a8d53269aee4\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m" Oct 30 11:14:17.950320 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.950284 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd4zc\" (UniqueName: \"kubernetes.io/projected/b1ed3274-9c04-4de7-b397-a8d53269aee4-kube-api-access-gd4zc\") pod \"korrel8r-5d45699dc4-qwh8m\" (UID: \"b1ed3274-9c04-4de7-b397-a8d53269aee4\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m" Oct 30 11:14:17.950320 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.950320 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1ed3274-9c04-4de7-b397-a8d53269aee4-config\") pod \"korrel8r-5d45699dc4-qwh8m\" (UID: \"b1ed3274-9c04-4de7-b397-a8d53269aee4\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m" Oct 30 11:14:17.950665 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.950345 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b1ed3274-9c04-4de7-b397-a8d53269aee4-tls\") pod \"korrel8r-5d45699dc4-qwh8m\" (UID: \"b1ed3274-9c04-4de7-b397-a8d53269aee4\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m" Oct 30 11:14:17.951095 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.951071 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1ed3274-9c04-4de7-b397-a8d53269aee4-config\") pod \"korrel8r-5d45699dc4-qwh8m\" (UID: \"b1ed3274-9c04-4de7-b397-a8d53269aee4\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m" Oct 30 11:14:17.952687 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.952670 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b1ed3274-9c04-4de7-b397-a8d53269aee4-tls\") pod \"korrel8r-5d45699dc4-qwh8m\" (UID: \"b1ed3274-9c04-4de7-b397-a8d53269aee4\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m" Oct 30 11:14:17.962968 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.962940 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:14:17.973343 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:17.973321 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd4zc\" (UniqueName: \"kubernetes.io/projected/b1ed3274-9c04-4de7-b397-a8d53269aee4-kube-api-access-gd4zc\") pod \"korrel8r-5d45699dc4-qwh8m\" (UID: \"b1ed3274-9c04-4de7-b397-a8d53269aee4\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m" Oct 30 11:14:18.123065 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:18.122987 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m" Oct 30 11:14:18.383602 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:18.381823 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:14:18.386046 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:18.386020 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:14:18.417901 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:18.417871 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:14:18.419684 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:18.419651 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m"] Oct 30 11:14:18.419956 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:14:18.419888 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ed3274_9c04_4de7_b397_a8d53269aee4.slice/crio-59c767fa8f80bd91ef01109ab3118b3e73e43b680aba965eabce5209cd9db06b WatchSource:0}: Error finding container 59c767fa8f80bd91ef01109ab3118b3e73e43b680aba965eabce5209cd9db06b: Status 404 returned error can't find the container with id 59c767fa8f80bd91ef01109ab3118b3e73e43b680aba965eabce5209cd9db06b Oct 30 11:14:18.454479 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:18.454423 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9239df1c-bd51-4e17-adec-696eb9987d52-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9239df1c-bd51-4e17-adec-696eb9987d52\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:14:18.454597 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:18.454546 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9239df1c-bd51-4e17-adec-696eb9987d52-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"9239df1c-bd51-4e17-adec-696eb9987d52\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:14:18.454691 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:18.454669 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2ch8\" (UniqueName: \"kubernetes.io/projected/9239df1c-bd51-4e17-adec-696eb9987d52-kube-api-access-k2ch8\") pod \"data-science-collector-collector-1\" (UID: \"9239df1c-bd51-4e17-adec-696eb9987d52\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:14:18.476303 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:18.476281 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-29k2n\"" Oct 30 11:14:18.476408 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:18.476317 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 11:14:18.476477 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:18.476420 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-0ed87044\"" Oct 30 11:14:18.487312 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:18.487290 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:14:18.556007 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:18.555975 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2ch8\" (UniqueName: \"kubernetes.io/projected/9239df1c-bd51-4e17-adec-696eb9987d52-kube-api-access-k2ch8\") pod \"data-science-collector-collector-1\" (UID: \"9239df1c-bd51-4e17-adec-696eb9987d52\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:14:18.556164 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:18.556033 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9239df1c-bd51-4e17-adec-696eb9987d52-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9239df1c-bd51-4e17-adec-696eb9987d52\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:14:18.556164 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:18.556061 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9239df1c-bd51-4e17-adec-696eb9987d52-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"9239df1c-bd51-4e17-adec-696eb9987d52\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:14:18.556696 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:18.556676 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9239df1c-bd51-4e17-adec-696eb9987d52-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"9239df1c-bd51-4e17-adec-696eb9987d52\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:14:18.558490 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:18.558420 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9239df1c-bd51-4e17-adec-696eb9987d52-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9239df1c-bd51-4e17-adec-696eb9987d52\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:14:18.579158 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:18.579122 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2ch8\" (UniqueName: \"kubernetes.io/projected/9239df1c-bd51-4e17-adec-696eb9987d52-kube-api-access-k2ch8\") pod \"data-science-collector-collector-1\" (UID: \"9239df1c-bd51-4e17-adec-696eb9987d52\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:14:18.681972 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:18.681889 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:14:18.703717 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:18.703691 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:14:18.996136 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:14:18.996103 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9239df1c_bd51_4e17_adec_696eb9987d52.slice/crio-24e845733f426cb5945141d6951a9c22093f9a9fe38fa0df1ac979a75345f4aa WatchSource:0}: Error finding container 24e845733f426cb5945141d6951a9c22093f9a9fe38fa0df1ac979a75345f4aa: Status 404 returned error can't find the container with id 24e845733f426cb5945141d6951a9c22093f9a9fe38fa0df1ac979a75345f4aa Oct 30 11:14:18.996462 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:18.996295 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:14:18.998295 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:18.998246 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:14:18.998382 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:18.998326 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:14:19.229722 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:19.229684 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m" event={"ID":"b1ed3274-9c04-4de7-b397-a8d53269aee4","Type":"ContainerStarted","Data":"c355ae9c6c1216ea83229b2909bc0e0370c79bb96aa53e5dda0ac6e0c31680b8"} Oct 30 11:14:19.229906 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:19.229752 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m" Oct 30 11:14:19.229906 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:19.229772 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m" event={"ID":"b1ed3274-9c04-4de7-b397-a8d53269aee4","Type":"ContainerStarted","Data":"59c767fa8f80bd91ef01109ab3118b3e73e43b680aba965eabce5209cd9db06b"} Oct 30 11:14:19.231203 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:19.231181 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"9239df1c-bd51-4e17-adec-696eb9987d52","Type":"ContainerStarted","Data":"1ab465a32a6b7225b4c4e1168fb9ac6c9b1cc781804db40011856fd24c944f65"} Oct 30 11:14:19.231315 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:19.231224 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"9239df1c-bd51-4e17-adec-696eb9987d52","Type":"ContainerStarted","Data":"24e845733f426cb5945141d6951a9c22093f9a9fe38fa0df1ac979a75345f4aa"} Oct 30 11:14:19.270576 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:19.270498 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m" podStartSLOduration=2.270485664 podStartE2EDuration="2.270485664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:14:19.269139123 +0000 UTC m=+1749.770518608" watchObservedRunningTime="2026-10-30 11:14:19.270485664 +0000 UTC m=+1749.771865149" Oct 30 11:14:19.311882 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:19.311828 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.311814776 podStartE2EDuration="1.311814776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:14:19.311303532 +0000 UTC m=+1749.812683039" watchObservedRunningTime="2026-10-30 11:14:19.311814776 +0000 UTC m=+1749.813194308" Oct 30 11:14:21.369935 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.369902 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:14:21.540097 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.540059 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:14:21.709315 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.709227 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:14:21.826682 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.826652 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6"] Oct 30 11:14:21.830226 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.830211 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:21.845459 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.845412 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 30 11:14:21.857341 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.857313 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-cabundle\"" Oct 30 11:14:21.857341 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.857313 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-dockercfg-hnf5m\"" Oct 30 11:14:21.857861 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.857392 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-mtls\"" Oct 30 11:14:21.857861 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.857409 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 30 11:14:21.857861 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.857486 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ca-bundle\"" Oct 30 11:14:21.857861 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.857636 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-tls\"" Oct 30 11:14:21.859442 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.859408 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6"] Oct 30 11:14:21.877379 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.877336 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:14:21.887340 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.887314 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/f16ad354-9fea-437a-ab8d-21555f055e68-tenant\") pod \"tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:21.887463 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.887355 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/f16ad354-9fea-437a-ab8d-21555f055e68-serving-certs\") pod \"tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:21.887463 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.887376 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7kld\" (UniqueName: \"kubernetes.io/projected/f16ad354-9fea-437a-ab8d-21555f055e68-kube-api-access-k7kld\") pod \"tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:21.887570 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.887463 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16ad354-9fea-437a-ab8d-21555f055e68-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:21.887570 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.887497 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f16ad354-9fea-437a-ab8d-21555f055e68-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:21.887636 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.887588 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/f16ad354-9fea-437a-ab8d-21555f055e68-rbac\") pod \"tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:21.887636 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.887607 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/f16ad354-9fea-437a-ab8d-21555f055e68-cabundle\") pod \"tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:21.988694 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.988661 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/f16ad354-9fea-437a-ab8d-21555f055e68-tenant\") pod \"tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:21.988890 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.988709 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/f16ad354-9fea-437a-ab8d-21555f055e68-serving-certs\") pod \"tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:21.988890 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.988728 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7kld\" (UniqueName: \"kubernetes.io/projected/f16ad354-9fea-437a-ab8d-21555f055e68-kube-api-access-k7kld\") pod \"tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:21.988890 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.988748 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16ad354-9fea-437a-ab8d-21555f055e68-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:21.988890 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.988768 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f16ad354-9fea-437a-ab8d-21555f055e68-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:21.988890 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.988822 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/f16ad354-9fea-437a-ab8d-21555f055e68-rbac\") pod \"tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:21.988890 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.988840 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/f16ad354-9fea-437a-ab8d-21555f055e68-cabundle\") pod \"tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:21.989608 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.989544 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/f16ad354-9fea-437a-ab8d-21555f055e68-rbac\") pod \"tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:21.989734 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.989610 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/f16ad354-9fea-437a-ab8d-21555f055e68-cabundle\") pod \"tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:21.989734 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.989609 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16ad354-9fea-437a-ab8d-21555f055e68-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:21.991293 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.991266 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/f16ad354-9fea-437a-ab8d-21555f055e68-tenant\") pod \"tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:21.991386 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.991339 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f16ad354-9fea-437a-ab8d-21555f055e68-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:21.991422 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:21.991392 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/f16ad354-9fea-437a-ab8d-21555f055e68-serving-certs\") pod \"tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:22.011513 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:22.011490 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7kld\" (UniqueName: \"kubernetes.io/projected/f16ad354-9fea-437a-ab8d-21555f055e68-kube-api-access-k7kld\") pod \"tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:22.144725 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:22.144691 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:22.470174 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:14:22.465934 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16ad354_9fea_437a_ab8d_21555f055e68.slice/crio-3de63ca634b5cbb7bea80365658565c8f9198ff87704b3dd276cf8c03812e334 WatchSource:0}: Error finding container 3de63ca634b5cbb7bea80365658565c8f9198ff87704b3dd276cf8c03812e334: Status 404 returned error can't find the container with id 3de63ca634b5cbb7bea80365658565c8f9198ff87704b3dd276cf8c03812e334 Oct 30 11:14:22.470174 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:22.466246 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6"] Oct 30 11:14:22.470550 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:22.470298 4289 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 11:14:23.251277 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:23.251240 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" event={"ID":"f16ad354-9fea-437a-ab8d-21555f055e68","Type":"ContainerStarted","Data":"3de63ca634b5cbb7bea80365658565c8f9198ff87704b3dd276cf8c03812e334"} Oct 30 11:14:23.720109 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:23.720007 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:14:23.885952 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:23.885918 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:14:24.502813 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:24.502773 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:14:25.260942 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:25.260907 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" event={"ID":"f16ad354-9fea-437a-ab8d-21555f055e68","Type":"ContainerStarted","Data":"8b0a4282f1010cb46fdecf707d8302a4504835e7790fe95c62b74dd806b0b726"} Oct 30 11:14:25.362000 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:25.361967 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6"] Oct 30 11:14:26.260113 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:14:26.260078 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:14:27.270926 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.270890 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" event={"ID":"f16ad354-9fea-437a-ab8d-21555f055e68","Type":"ContainerStarted","Data":"59ecc23f997fef13ee2c7c92e01b6e80b882f1e9425c1e1149a31a9f660f6f50"} Oct 30 11:14:27.271376 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.270990 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" podUID="f16ad354-9fea-437a-ab8d-21555f055e68" containerName="tempo-gateway" containerID="cri-o://8b0a4282f1010cb46fdecf707d8302a4504835e7790fe95c62b74dd806b0b726" gracePeriod=30 Oct 30 11:14:27.271376 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.271028 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" podUID="f16ad354-9fea-437a-ab8d-21555f055e68" containerName="tempo-gateway-opa" containerID="cri-o://59ecc23f997fef13ee2c7c92e01b6e80b882f1e9425c1e1149a31a9f660f6f50" gracePeriod=30 Oct 30 11:14:27.271376 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.271119 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:27.272379 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.272352 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" podUID="f16ad354-9fea-437a-ab8d-21555f055e68" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.134.0.62:8083/ready\": dial tcp 10.134.0.62:8083: connect: connection refused" Oct 30 11:14:27.339224 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.339180 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" podStartSLOduration=2.5851502870000003 podStartE2EDuration="6.339167074s" totalImagesPullingTime="3.754016787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:14:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:14:27.338157296 +0000 UTC m=+1757.839536782" watchObservedRunningTime="2026-10-30 11:14:27.339167074 +0000 UTC m=+1757.840546602" Oct 30 11:14:27.613920 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.613901 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:27.740798 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.740767 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f16ad354-9fea-437a-ab8d-21555f055e68-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16ad354-9fea-437a-ab8d-21555f055e68-tempo-data-science-tempostack-ca-bundle\") pod \"f16ad354-9fea-437a-ab8d-21555f055e68\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " Oct 30 11:14:27.740992 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.740805 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f16ad354-9fea-437a-ab8d-21555f055e68-serving-certs\" (UniqueName: \"kubernetes.io/secret/f16ad354-9fea-437a-ab8d-21555f055e68-serving-certs\") pod \"f16ad354-9fea-437a-ab8d-21555f055e68\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " Oct 30 11:14:27.740992 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.740834 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f16ad354-9fea-437a-ab8d-21555f055e68-tenant\" (UniqueName: \"kubernetes.io/secret/f16ad354-9fea-437a-ab8d-21555f055e68-tenant\") pod \"f16ad354-9fea-437a-ab8d-21555f055e68\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " Oct 30 11:14:27.740992 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.740855 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f16ad354-9fea-437a-ab8d-21555f055e68-rbac\" (UniqueName: \"kubernetes.io/configmap/f16ad354-9fea-437a-ab8d-21555f055e68-rbac\") pod \"f16ad354-9fea-437a-ab8d-21555f055e68\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " Oct 30 11:14:27.740992 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.740896 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f16ad354-9fea-437a-ab8d-21555f055e68-tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f16ad354-9fea-437a-ab8d-21555f055e68-tempo-data-science-tempostack-gateway-mtls\") pod \"f16ad354-9fea-437a-ab8d-21555f055e68\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " Oct 30 11:14:27.740992 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.740941 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f16ad354-9fea-437a-ab8d-21555f055e68-kube-api-access-k7kld\" (UniqueName: \"kubernetes.io/projected/f16ad354-9fea-437a-ab8d-21555f055e68-kube-api-access-k7kld\") pod \"f16ad354-9fea-437a-ab8d-21555f055e68\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " Oct 30 11:14:27.741281 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.741011 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f16ad354-9fea-437a-ab8d-21555f055e68-cabundle\" (UniqueName: \"kubernetes.io/configmap/f16ad354-9fea-437a-ab8d-21555f055e68-cabundle\") pod \"f16ad354-9fea-437a-ab8d-21555f055e68\" (UID: \"f16ad354-9fea-437a-ab8d-21555f055e68\") " Oct 30 11:14:27.741281 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.741180 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16ad354-9fea-437a-ab8d-21555f055e68-tempo-data-science-tempostack-ca-bundle" pod "f16ad354-9fea-437a-ab8d-21555f055e68" (UID: "f16ad354-9fea-437a-ab8d-21555f055e68"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:14:27.741281 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.741272 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16ad354-9fea-437a-ab8d-21555f055e68-rbac" pod "f16ad354-9fea-437a-ab8d-21555f055e68" (UID: "f16ad354-9fea-437a-ab8d-21555f055e68"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:14:27.741465 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.741416 4289 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16ad354-9fea-437a-ab8d-21555f055e68-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:14:27.741532 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.741463 4289 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/f16ad354-9fea-437a-ab8d-21555f055e68-rbac\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:14:27.741789 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.741735 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16ad354-9fea-437a-ab8d-21555f055e68-cabundle" pod "f16ad354-9fea-437a-ab8d-21555f055e68" (UID: "f16ad354-9fea-437a-ab8d-21555f055e68"). InnerVolumeSpecName "cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:14:27.743205 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.743178 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f16ad354-9fea-437a-ab8d-21555f055e68-serving-certs" pod "f16ad354-9fea-437a-ab8d-21555f055e68" (UID: "f16ad354-9fea-437a-ab8d-21555f055e68"). InnerVolumeSpecName "serving-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:14:27.743285 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.743258 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f16ad354-9fea-437a-ab8d-21555f055e68-tempo-data-science-tempostack-gateway-mtls" pod "f16ad354-9fea-437a-ab8d-21555f055e68" (UID: "f16ad354-9fea-437a-ab8d-21555f055e68"). InnerVolumeSpecName "tempo-data-science-tempostack-gateway-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:14:27.743659 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.743634 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16ad354-9fea-437a-ab8d-21555f055e68-kube-api-access-k7kld" pod "f16ad354-9fea-437a-ab8d-21555f055e68" (UID: "f16ad354-9fea-437a-ab8d-21555f055e68"). InnerVolumeSpecName "kube-api-access-k7kld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:14:27.746863 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.746845 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f16ad354-9fea-437a-ab8d-21555f055e68-tenant" pod "f16ad354-9fea-437a-ab8d-21555f055e68" (UID: "f16ad354-9fea-437a-ab8d-21555f055e68"). InnerVolumeSpecName "tenant". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:14:27.842475 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.842376 4289 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f16ad354-9fea-437a-ab8d-21555f055e68-tempo-data-science-tempostack-gateway-mtls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:14:27.842475 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.842414 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7kld\" (UniqueName: \"kubernetes.io/projected/f16ad354-9fea-437a-ab8d-21555f055e68-kube-api-access-k7kld\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:14:27.842475 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.842442 4289 reconciler_common.go:299] "Volume detached for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/f16ad354-9fea-437a-ab8d-21555f055e68-cabundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:14:27.842475 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.842453 4289 reconciler_common.go:299] "Volume detached for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/f16ad354-9fea-437a-ab8d-21555f055e68-serving-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:14:27.842475 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:27.842461 4289 reconciler_common.go:299] "Volume detached for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/f16ad354-9fea-437a-ab8d-21555f055e68-tenant\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:14:28.276467 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:28.276421 4289 generic.go:356] "Generic (PLEG): container finished" podID="f16ad354-9fea-437a-ab8d-21555f055e68" containerID="59ecc23f997fef13ee2c7c92e01b6e80b882f1e9425c1e1149a31a9f660f6f50" exitCode=2 Oct 30 11:14:28.276467 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:28.276464 4289 generic.go:356] "Generic (PLEG): container finished" podID="f16ad354-9fea-437a-ab8d-21555f055e68" containerID="8b0a4282f1010cb46fdecf707d8302a4504835e7790fe95c62b74dd806b0b726" exitCode=0 Oct 30 11:14:28.276918 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:28.276488 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" Oct 30 11:14:28.276918 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:28.276503 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" event={"ID":"f16ad354-9fea-437a-ab8d-21555f055e68","Type":"ContainerDied","Data":"59ecc23f997fef13ee2c7c92e01b6e80b882f1e9425c1e1149a31a9f660f6f50"} Oct 30 11:14:28.276918 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:28.276561 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" event={"ID":"f16ad354-9fea-437a-ab8d-21555f055e68","Type":"ContainerDied","Data":"8b0a4282f1010cb46fdecf707d8302a4504835e7790fe95c62b74dd806b0b726"} Oct 30 11:14:28.276918 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:28.276582 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6" event={"ID":"f16ad354-9fea-437a-ab8d-21555f055e68","Type":"ContainerDied","Data":"3de63ca634b5cbb7bea80365658565c8f9198ff87704b3dd276cf8c03812e334"} Oct 30 11:14:28.276918 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:28.276608 4289 scope.go:122] "RemoveContainer" containerID="59ecc23f997fef13ee2c7c92e01b6e80b882f1e9425c1e1149a31a9f660f6f50" Oct 30 11:14:28.284982 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:28.284967 4289 scope.go:122] "RemoveContainer" containerID="8b0a4282f1010cb46fdecf707d8302a4504835e7790fe95c62b74dd806b0b726" Oct 30 11:14:28.292512 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:28.292494 4289 scope.go:122] "RemoveContainer" containerID="59ecc23f997fef13ee2c7c92e01b6e80b882f1e9425c1e1149a31a9f660f6f50" Oct 30 11:14:28.292762 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:14:28.292746 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ecc23f997fef13ee2c7c92e01b6e80b882f1e9425c1e1149a31a9f660f6f50\": container with ID starting with 59ecc23f997fef13ee2c7c92e01b6e80b882f1e9425c1e1149a31a9f660f6f50 not found: ID does not exist" containerID="59ecc23f997fef13ee2c7c92e01b6e80b882f1e9425c1e1149a31a9f660f6f50" Oct 30 11:14:28.292801 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:28.292771 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ecc23f997fef13ee2c7c92e01b6e80b882f1e9425c1e1149a31a9f660f6f50"} err="failed to get container status \"59ecc23f997fef13ee2c7c92e01b6e80b882f1e9425c1e1149a31a9f660f6f50\": rpc error: code = NotFound desc = could not find container \"59ecc23f997fef13ee2c7c92e01b6e80b882f1e9425c1e1149a31a9f660f6f50\": container with ID starting with 59ecc23f997fef13ee2c7c92e01b6e80b882f1e9425c1e1149a31a9f660f6f50 not found: ID does not exist" Oct 30 11:14:28.292801 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:28.292787 4289 scope.go:122] "RemoveContainer" containerID="8b0a4282f1010cb46fdecf707d8302a4504835e7790fe95c62b74dd806b0b726" Oct 30 11:14:28.293001 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:14:28.292982 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0a4282f1010cb46fdecf707d8302a4504835e7790fe95c62b74dd806b0b726\": container with ID starting with 8b0a4282f1010cb46fdecf707d8302a4504835e7790fe95c62b74dd806b0b726 not found: ID does not exist" containerID="8b0a4282f1010cb46fdecf707d8302a4504835e7790fe95c62b74dd806b0b726" Oct 30 11:14:28.293039 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:28.293008 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0a4282f1010cb46fdecf707d8302a4504835e7790fe95c62b74dd806b0b726"} err="failed to get container status \"8b0a4282f1010cb46fdecf707d8302a4504835e7790fe95c62b74dd806b0b726\": rpc error: code = NotFound desc = could not find container \"8b0a4282f1010cb46fdecf707d8302a4504835e7790fe95c62b74dd806b0b726\": container with ID starting with 8b0a4282f1010cb46fdecf707d8302a4504835e7790fe95c62b74dd806b0b726 not found: ID does not exist" Oct 30 11:14:28.293039 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:28.293024 4289 scope.go:122] "RemoveContainer" containerID="59ecc23f997fef13ee2c7c92e01b6e80b882f1e9425c1e1149a31a9f660f6f50" Oct 30 11:14:28.293225 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:28.293209 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ecc23f997fef13ee2c7c92e01b6e80b882f1e9425c1e1149a31a9f660f6f50"} err="failed to get container status \"59ecc23f997fef13ee2c7c92e01b6e80b882f1e9425c1e1149a31a9f660f6f50\": rpc error: code = NotFound desc = could not find container \"59ecc23f997fef13ee2c7c92e01b6e80b882f1e9425c1e1149a31a9f660f6f50\": container with ID starting with 59ecc23f997fef13ee2c7c92e01b6e80b882f1e9425c1e1149a31a9f660f6f50 not found: ID does not exist" Oct 30 11:14:28.293261 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:28.293225 4289 scope.go:122] "RemoveContainer" containerID="8b0a4282f1010cb46fdecf707d8302a4504835e7790fe95c62b74dd806b0b726" Oct 30 11:14:28.293424 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:28.293404 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0a4282f1010cb46fdecf707d8302a4504835e7790fe95c62b74dd806b0b726"} err="failed to get container status \"8b0a4282f1010cb46fdecf707d8302a4504835e7790fe95c62b74dd806b0b726\": rpc error: code = NotFound desc = could not find container \"8b0a4282f1010cb46fdecf707d8302a4504835e7790fe95c62b74dd806b0b726\": container with ID starting with 8b0a4282f1010cb46fdecf707d8302a4504835e7790fe95c62b74dd806b0b726 not found: ID does not exist" Oct 30 11:14:28.340337 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:28.340305 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6"] Oct 30 11:14:28.351005 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:28.350978 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-867b59b7bf-h6ql6"] Oct 30 11:14:29.624675 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:29.624643 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f16ad354-9fea-437a-ab8d-21555f055e68" path="/var/lib/kubelet/pods/f16ad354-9fea-437a-ab8d-21555f055e68/volumes" Oct 30 11:14:30.237584 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:30.237549 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m" Oct 30 11:14:52.387122 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:52.387077 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:14:56.258447 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:14:56.258336 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:14:56.817315 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:14:56.817283 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:15:26.257620 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:15:26.257577 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:15:26.597068 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:26.596969 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:15:26.777071 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:26.777041 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:15:27.170487 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.170453 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:15:27.170759 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.170634 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="9239df1c-bd51-4e17-adec-696eb9987d52" containerName="otc-container" containerID="cri-o://1ab465a32a6b7225b4c4e1168fb9ac6c9b1cc781804db40011856fd24c944f65" gracePeriod=30 Oct 30 11:15:27.344623 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.344546 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:15:27.416862 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.416838 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:15:27.516997 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.516961 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:15:27.522248 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.522223 4289 generic.go:356] "Generic (PLEG): container finished" podID="9239df1c-bd51-4e17-adec-696eb9987d52" containerID="1ab465a32a6b7225b4c4e1168fb9ac6c9b1cc781804db40011856fd24c944f65" exitCode=0 Oct 30 11:15:27.522362 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.522274 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:15:27.522362 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.522280 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"9239df1c-bd51-4e17-adec-696eb9987d52","Type":"ContainerDied","Data":"1ab465a32a6b7225b4c4e1168fb9ac6c9b1cc781804db40011856fd24c944f65"} Oct 30 11:15:27.522362 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.522328 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"9239df1c-bd51-4e17-adec-696eb9987d52","Type":"ContainerDied","Data":"24e845733f426cb5945141d6951a9c22093f9a9fe38fa0df1ac979a75345f4aa"} Oct 30 11:15:27.522362 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.522350 4289 scope.go:122] "RemoveContainer" containerID="1ab465a32a6b7225b4c4e1168fb9ac6c9b1cc781804db40011856fd24c944f65" Oct 30 11:15:27.528589 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.528567 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9239df1c-bd51-4e17-adec-696eb9987d52-kube-api-access-k2ch8\" (UniqueName: \"kubernetes.io/projected/9239df1c-bd51-4e17-adec-696eb9987d52-kube-api-access-k2ch8\") pod \"9239df1c-bd51-4e17-adec-696eb9987d52\" (UID: \"9239df1c-bd51-4e17-adec-696eb9987d52\") " Oct 30 11:15:27.528726 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.528633 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9239df1c-bd51-4e17-adec-696eb9987d52-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9239df1c-bd51-4e17-adec-696eb9987d52-monitoring-tls-certs\") pod \"9239df1c-bd51-4e17-adec-696eb9987d52\" (UID: \"9239df1c-bd51-4e17-adec-696eb9987d52\") " Oct 30 11:15:27.528726 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.528656 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9239df1c-bd51-4e17-adec-696eb9987d52-otc-internal\" (UniqueName: \"kubernetes.io/configmap/9239df1c-bd51-4e17-adec-696eb9987d52-otc-internal\") pod \"9239df1c-bd51-4e17-adec-696eb9987d52\" (UID: \"9239df1c-bd51-4e17-adec-696eb9987d52\") " Oct 30 11:15:27.529072 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.529043 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9239df1c-bd51-4e17-adec-696eb9987d52-otc-internal" pod "9239df1c-bd51-4e17-adec-696eb9987d52" (UID: "9239df1c-bd51-4e17-adec-696eb9987d52"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:15:27.530343 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.530323 4289 scope.go:122] "RemoveContainer" containerID="1ab465a32a6b7225b4c4e1168fb9ac6c9b1cc781804db40011856fd24c944f65" Oct 30 11:15:27.530612 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.530576 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9239df1c-bd51-4e17-adec-696eb9987d52-kube-api-access-k2ch8" pod "9239df1c-bd51-4e17-adec-696eb9987d52" (UID: "9239df1c-bd51-4e17-adec-696eb9987d52"). InnerVolumeSpecName "kube-api-access-k2ch8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:15:27.530612 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.530594 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9239df1c-bd51-4e17-adec-696eb9987d52-monitoring-tls-certs" pod "9239df1c-bd51-4e17-adec-696eb9987d52" (UID: "9239df1c-bd51-4e17-adec-696eb9987d52"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:15:27.530741 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:15:27.530633 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab465a32a6b7225b4c4e1168fb9ac6c9b1cc781804db40011856fd24c944f65\": container with ID starting with 1ab465a32a6b7225b4c4e1168fb9ac6c9b1cc781804db40011856fd24c944f65 not found: ID does not exist" containerID="1ab465a32a6b7225b4c4e1168fb9ac6c9b1cc781804db40011856fd24c944f65" Oct 30 11:15:27.530741 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.530662 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab465a32a6b7225b4c4e1168fb9ac6c9b1cc781804db40011856fd24c944f65"} err="failed to get container status \"1ab465a32a6b7225b4c4e1168fb9ac6c9b1cc781804db40011856fd24c944f65\": rpc error: code = NotFound desc = could not find container \"1ab465a32a6b7225b4c4e1168fb9ac6c9b1cc781804db40011856fd24c944f65\": container with ID starting with 1ab465a32a6b7225b4c4e1168fb9ac6c9b1cc781804db40011856fd24c944f65 not found: ID does not exist" Oct 30 11:15:27.629386 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.629356 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2ch8\" (UniqueName: \"kubernetes.io/projected/9239df1c-bd51-4e17-adec-696eb9987d52-kube-api-access-k2ch8\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:15:27.629386 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.629381 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9239df1c-bd51-4e17-adec-696eb9987d52-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:15:27.629386 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.629390 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9239df1c-bd51-4e17-adec-696eb9987d52-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:15:27.860030 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.859948 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:15:27.871940 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.871913 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:15:27.902230 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.902201 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:15:27.902556 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.902542 4289 cpu_manager.go:414] "containerMap: removing container" podUID="f16ad354-9fea-437a-ab8d-21555f055e68" containerName="tempo-gateway-opa" Oct 30 11:15:27.902601 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.902557 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f16ad354-9fea-437a-ab8d-21555f055e68" containerName="tempo-gateway-opa" Oct 30 11:15:27.902601 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.902576 4289 cpu_manager.go:414] "containerMap: removing container" podUID="9239df1c-bd51-4e17-adec-696eb9987d52" containerName="otc-container" Oct 30 11:15:27.902601 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.902582 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9239df1c-bd51-4e17-adec-696eb9987d52" containerName="otc-container" Oct 30 11:15:27.902700 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.902603 4289 cpu_manager.go:414] "containerMap: removing container" podUID="f16ad354-9fea-437a-ab8d-21555f055e68" containerName="tempo-gateway" Oct 30 11:15:27.902700 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.902608 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f16ad354-9fea-437a-ab8d-21555f055e68" containerName="tempo-gateway" Oct 30 11:15:27.902700 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.902659 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="f16ad354-9fea-437a-ab8d-21555f055e68" containerName="tempo-gateway" Oct 30 11:15:27.902700 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.902671 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="f16ad354-9fea-437a-ab8d-21555f055e68" containerName="tempo-gateway-opa" Oct 30 11:15:27.902700 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.902681 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="9239df1c-bd51-4e17-adec-696eb9987d52" containerName="otc-container" Oct 30 11:15:27.906853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.906834 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:15:27.924921 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.924899 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:15:27.950413 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.950389 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-0ed87044\"" Oct 30 11:15:27.966821 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.966794 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-29k2n\"" Oct 30 11:15:27.966955 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:27.966885 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 11:15:28.033307 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:28.033275 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9907347d-d4b2-47e7-9754-5f9a781ade37-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9907347d-d4b2-47e7-9754-5f9a781ade37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:15:28.033456 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:28.033366 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqbqc\" (UniqueName: \"kubernetes.io/projected/9907347d-d4b2-47e7-9754-5f9a781ade37-kube-api-access-sqbqc\") pod \"data-science-collector-collector-1\" (UID: \"9907347d-d4b2-47e7-9754-5f9a781ade37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:15:28.033456 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:28.033415 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9907347d-d4b2-47e7-9754-5f9a781ade37-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"9907347d-d4b2-47e7-9754-5f9a781ade37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:15:28.133846 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:28.133758 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9907347d-d4b2-47e7-9754-5f9a781ade37-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9907347d-d4b2-47e7-9754-5f9a781ade37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:15:28.134011 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:28.133858 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqbqc\" (UniqueName: \"kubernetes.io/projected/9907347d-d4b2-47e7-9754-5f9a781ade37-kube-api-access-sqbqc\") pod \"data-science-collector-collector-1\" (UID: \"9907347d-d4b2-47e7-9754-5f9a781ade37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:15:28.134011 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:28.133890 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9907347d-d4b2-47e7-9754-5f9a781ade37-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"9907347d-d4b2-47e7-9754-5f9a781ade37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:15:28.134543 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:28.134519 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9907347d-d4b2-47e7-9754-5f9a781ade37-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"9907347d-d4b2-47e7-9754-5f9a781ade37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:15:28.136091 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:28.136073 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9907347d-d4b2-47e7-9754-5f9a781ade37-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9907347d-d4b2-47e7-9754-5f9a781ade37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:15:28.156847 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:28.156820 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqbqc\" (UniqueName: \"kubernetes.io/projected/9907347d-d4b2-47e7-9754-5f9a781ade37-kube-api-access-sqbqc\") pod \"data-science-collector-collector-1\" (UID: \"9907347d-d4b2-47e7-9754-5f9a781ade37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:15:28.218814 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:28.218774 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:15:28.511416 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:28.511323 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:15:28.511857 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:15:28.511544 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9907347d_d4b2_47e7_9754_5f9a781ade37.slice/crio-819e20dc422a4c41cb644ab544dec8ecc2dbc255179bc8cbe84fbde1062dd6be WatchSource:0}: Error finding container 819e20dc422a4c41cb644ab544dec8ecc2dbc255179bc8cbe84fbde1062dd6be: Status 404 returned error can't find the container with id 819e20dc422a4c41cb644ab544dec8ecc2dbc255179bc8cbe84fbde1062dd6be Oct 30 11:15:28.513921 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:28.513877 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:15:28.514008 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:28.513959 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:15:28.529860 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:28.529824 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"9907347d-d4b2-47e7-9754-5f9a781ade37","Type":"ContainerStarted","Data":"819e20dc422a4c41cb644ab544dec8ecc2dbc255179bc8cbe84fbde1062dd6be"} Oct 30 11:15:29.535893 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:29.535859 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"9907347d-d4b2-47e7-9754-5f9a781ade37","Type":"ContainerStarted","Data":"e9f0499cb8a9672c40423ef37a90a1ffc15bd2d060a4aa706bd17304a2eaaee9"} Oct 30 11:15:29.578366 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:29.578317 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.578303831 podStartE2EDuration="2.578303831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:15:29.576622601 +0000 UTC m=+1820.078002113" watchObservedRunningTime="2026-10-30 11:15:29.578303831 +0000 UTC m=+1820.079683315" Oct 30 11:15:29.624682 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:29.624650 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9239df1c-bd51-4e17-adec-696eb9987d52" path="/var/lib/kubelet/pods/9239df1c-bd51-4e17-adec-696eb9987d52/volumes" Oct 30 11:15:31.290613 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.290580 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m"] Oct 30 11:15:31.291046 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.290812 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m" podUID="b1ed3274-9c04-4de7-b397-a8d53269aee4" containerName="korrel8r" containerID="cri-o://c355ae9c6c1216ea83229b2909bc0e0370c79bb96aa53e5dda0ac6e0c31680b8" gracePeriod=30 Oct 30 11:15:31.542529 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.542459 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m" Oct 30 11:15:31.546252 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.546225 4289 generic.go:356] "Generic (PLEG): container finished" podID="b1ed3274-9c04-4de7-b397-a8d53269aee4" containerID="c355ae9c6c1216ea83229b2909bc0e0370c79bb96aa53e5dda0ac6e0c31680b8" exitCode=0 Oct 30 11:15:31.546392 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.546260 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m" event={"ID":"b1ed3274-9c04-4de7-b397-a8d53269aee4","Type":"ContainerDied","Data":"c355ae9c6c1216ea83229b2909bc0e0370c79bb96aa53e5dda0ac6e0c31680b8"} Oct 30 11:15:31.546392 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.546268 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m" Oct 30 11:15:31.546392 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.546295 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m" event={"ID":"b1ed3274-9c04-4de7-b397-a8d53269aee4","Type":"ContainerDied","Data":"59c767fa8f80bd91ef01109ab3118b3e73e43b680aba965eabce5209cd9db06b"} Oct 30 11:15:31.546392 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.546322 4289 scope.go:122] "RemoveContainer" containerID="c355ae9c6c1216ea83229b2909bc0e0370c79bb96aa53e5dda0ac6e0c31680b8" Oct 30 11:15:31.554055 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.554027 4289 scope.go:122] "RemoveContainer" containerID="c355ae9c6c1216ea83229b2909bc0e0370c79bb96aa53e5dda0ac6e0c31680b8" Oct 30 11:15:31.554334 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:15:31.554314 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c355ae9c6c1216ea83229b2909bc0e0370c79bb96aa53e5dda0ac6e0c31680b8\": container with ID starting with c355ae9c6c1216ea83229b2909bc0e0370c79bb96aa53e5dda0ac6e0c31680b8 not found: ID does not exist" containerID="c355ae9c6c1216ea83229b2909bc0e0370c79bb96aa53e5dda0ac6e0c31680b8" Oct 30 11:15:31.554403 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.554347 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c355ae9c6c1216ea83229b2909bc0e0370c79bb96aa53e5dda0ac6e0c31680b8"} err="failed to get container status \"c355ae9c6c1216ea83229b2909bc0e0370c79bb96aa53e5dda0ac6e0c31680b8\": rpc error: code = NotFound desc = could not find container \"c355ae9c6c1216ea83229b2909bc0e0370c79bb96aa53e5dda0ac6e0c31680b8\": container with ID starting with c355ae9c6c1216ea83229b2909bc0e0370c79bb96aa53e5dda0ac6e0c31680b8 not found: ID does not exist" Oct 30 11:15:31.664523 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.664489 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1ed3274-9c04-4de7-b397-a8d53269aee4-tls\" (UniqueName: \"kubernetes.io/secret/b1ed3274-9c04-4de7-b397-a8d53269aee4-tls\") pod \"b1ed3274-9c04-4de7-b397-a8d53269aee4\" (UID: \"b1ed3274-9c04-4de7-b397-a8d53269aee4\") " Oct 30 11:15:31.664723 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.664548 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1ed3274-9c04-4de7-b397-a8d53269aee4-config\" (UniqueName: \"kubernetes.io/configmap/b1ed3274-9c04-4de7-b397-a8d53269aee4-config\") pod \"b1ed3274-9c04-4de7-b397-a8d53269aee4\" (UID: \"b1ed3274-9c04-4de7-b397-a8d53269aee4\") " Oct 30 11:15:31.664723 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.664634 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1ed3274-9c04-4de7-b397-a8d53269aee4-kube-api-access-gd4zc\" (UniqueName: \"kubernetes.io/projected/b1ed3274-9c04-4de7-b397-a8d53269aee4-kube-api-access-gd4zc\") pod \"b1ed3274-9c04-4de7-b397-a8d53269aee4\" (UID: \"b1ed3274-9c04-4de7-b397-a8d53269aee4\") " Oct 30 11:15:31.664970 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.664944 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ed3274-9c04-4de7-b397-a8d53269aee4-config" pod "b1ed3274-9c04-4de7-b397-a8d53269aee4" (UID: "b1ed3274-9c04-4de7-b397-a8d53269aee4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:15:31.669019 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.668990 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1ed3274-9c04-4de7-b397-a8d53269aee4-tls" pod "b1ed3274-9c04-4de7-b397-a8d53269aee4" (UID: "b1ed3274-9c04-4de7-b397-a8d53269aee4"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:15:31.670844 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.670812 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ed3274-9c04-4de7-b397-a8d53269aee4-kube-api-access-gd4zc" pod "b1ed3274-9c04-4de7-b397-a8d53269aee4" (UID: "b1ed3274-9c04-4de7-b397-a8d53269aee4"). InnerVolumeSpecName "kube-api-access-gd4zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:15:31.693723 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.693698 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:15:31.693898 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.693878 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="9907347d-d4b2-47e7-9754-5f9a781ade37" containerName="otc-container" containerID="cri-o://e9f0499cb8a9672c40423ef37a90a1ffc15bd2d060a4aa706bd17304a2eaaee9" gracePeriod=30 Oct 30 11:15:31.765262 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.765230 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1ed3274-9c04-4de7-b397-a8d53269aee4-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:15:31.765262 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.765257 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd4zc\" (UniqueName: \"kubernetes.io/projected/b1ed3274-9c04-4de7-b397-a8d53269aee4-kube-api-access-gd4zc\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:15:31.765262 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.765267 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b1ed3274-9c04-4de7-b397-a8d53269aee4-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:15:31.783406 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.783372 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:15:31.907967 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.907932 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m"] Oct 30 11:15:31.919736 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.919710 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-qwh8m"] Oct 30 11:15:31.931662 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.931641 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:15:31.959353 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:31.959324 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:15:32.066997 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:32.066922 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9907347d-d4b2-47e7-9754-5f9a781ade37-kube-api-access-sqbqc\" (UniqueName: \"kubernetes.io/projected/9907347d-d4b2-47e7-9754-5f9a781ade37-kube-api-access-sqbqc\") pod \"9907347d-d4b2-47e7-9754-5f9a781ade37\" (UID: \"9907347d-d4b2-47e7-9754-5f9a781ade37\") " Oct 30 11:15:32.066997 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:32.066983 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9907347d-d4b2-47e7-9754-5f9a781ade37-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9907347d-d4b2-47e7-9754-5f9a781ade37-monitoring-tls-certs\") pod \"9907347d-d4b2-47e7-9754-5f9a781ade37\" (UID: \"9907347d-d4b2-47e7-9754-5f9a781ade37\") " Oct 30 11:15:32.067181 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:32.067022 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9907347d-d4b2-47e7-9754-5f9a781ade37-otc-internal\" (UniqueName: \"kubernetes.io/configmap/9907347d-d4b2-47e7-9754-5f9a781ade37-otc-internal\") pod \"9907347d-d4b2-47e7-9754-5f9a781ade37\" (UID: \"9907347d-d4b2-47e7-9754-5f9a781ade37\") " Oct 30 11:15:32.067471 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:32.067423 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9907347d-d4b2-47e7-9754-5f9a781ade37-otc-internal" pod "9907347d-d4b2-47e7-9754-5f9a781ade37" (UID: "9907347d-d4b2-47e7-9754-5f9a781ade37"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:15:32.069073 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:32.069052 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9907347d-d4b2-47e7-9754-5f9a781ade37-monitoring-tls-certs" pod "9907347d-d4b2-47e7-9754-5f9a781ade37" (UID: "9907347d-d4b2-47e7-9754-5f9a781ade37"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:15:32.069137 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:32.069089 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9907347d-d4b2-47e7-9754-5f9a781ade37-kube-api-access-sqbqc" pod "9907347d-d4b2-47e7-9754-5f9a781ade37" (UID: "9907347d-d4b2-47e7-9754-5f9a781ade37"). InnerVolumeSpecName "kube-api-access-sqbqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:15:32.130201 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:32.130162 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:15:32.168120 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:32.168084 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqbqc\" (UniqueName: \"kubernetes.io/projected/9907347d-d4b2-47e7-9754-5f9a781ade37-kube-api-access-sqbqc\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:15:32.168120 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:32.168121 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9907347d-d4b2-47e7-9754-5f9a781ade37-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:15:32.168368 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:32.168137 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9907347d-d4b2-47e7-9754-5f9a781ade37-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:15:32.309618 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:32.309585 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:15:32.480275 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:32.480242 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:15:32.553889 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:32.553853 4289 generic.go:356] "Generic (PLEG): container finished" podID="9907347d-d4b2-47e7-9754-5f9a781ade37" containerID="e9f0499cb8a9672c40423ef37a90a1ffc15bd2d060a4aa706bd17304a2eaaee9" exitCode=0 Oct 30 11:15:32.554074 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:32.553917 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"9907347d-d4b2-47e7-9754-5f9a781ade37","Type":"ContainerDied","Data":"e9f0499cb8a9672c40423ef37a90a1ffc15bd2d060a4aa706bd17304a2eaaee9"} Oct 30 11:15:32.554074 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:32.553949 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:15:32.554074 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:32.553980 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"9907347d-d4b2-47e7-9754-5f9a781ade37","Type":"ContainerDied","Data":"819e20dc422a4c41cb644ab544dec8ecc2dbc255179bc8cbe84fbde1062dd6be"} Oct 30 11:15:32.554074 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:32.554006 4289 scope.go:122] "RemoveContainer" containerID="e9f0499cb8a9672c40423ef37a90a1ffc15bd2d060a4aa706bd17304a2eaaee9" Oct 30 11:15:32.562504 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:32.562483 4289 scope.go:122] "RemoveContainer" containerID="e9f0499cb8a9672c40423ef37a90a1ffc15bd2d060a4aa706bd17304a2eaaee9" Oct 30 11:15:32.562780 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:15:32.562758 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9f0499cb8a9672c40423ef37a90a1ffc15bd2d060a4aa706bd17304a2eaaee9\": container with ID starting with e9f0499cb8a9672c40423ef37a90a1ffc15bd2d060a4aa706bd17304a2eaaee9 not found: ID does not exist" containerID="e9f0499cb8a9672c40423ef37a90a1ffc15bd2d060a4aa706bd17304a2eaaee9" Oct 30 11:15:32.562847 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:32.562791 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9f0499cb8a9672c40423ef37a90a1ffc15bd2d060a4aa706bd17304a2eaaee9"} err="failed to get container status \"e9f0499cb8a9672c40423ef37a90a1ffc15bd2d060a4aa706bd17304a2eaaee9\": rpc error: code = NotFound desc = could not find container \"e9f0499cb8a9672c40423ef37a90a1ffc15bd2d060a4aa706bd17304a2eaaee9\": container with ID starting with e9f0499cb8a9672c40423ef37a90a1ffc15bd2d060a4aa706bd17304a2eaaee9 not found: ID does not exist" Oct 30 11:15:32.610725 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:32.610694 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:15:32.621392 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:32.621344 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:15:33.624977 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:33.624943 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9907347d-d4b2-47e7-9754-5f9a781ade37" path="/var/lib/kubelet/pods/9907347d-d4b2-47e7-9754-5f9a781ade37/volumes" Oct 30 11:15:33.625508 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:33.625486 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1ed3274-9c04-4de7-b397-a8d53269aee4" path="/var/lib/kubelet/pods/b1ed3274-9c04-4de7-b397-a8d53269aee4/volumes" Oct 30 11:15:56.256882 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:15:56.256798 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:15:56.953865 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:15:56.953825 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:16:00.266228 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:00.266187 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:16:02.542119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:02.542078 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:16:17.889288 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:17.889256 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6"] Oct 30 11:16:17.889678 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:17.889578 4289 cpu_manager.go:414] "containerMap: removing container" podUID="b1ed3274-9c04-4de7-b397-a8d53269aee4" containerName="korrel8r" Oct 30 11:16:17.889678 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:17.889589 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1ed3274-9c04-4de7-b397-a8d53269aee4" containerName="korrel8r" Oct 30 11:16:17.889678 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:17.889612 4289 cpu_manager.go:414] "containerMap: removing container" podUID="9907347d-d4b2-47e7-9754-5f9a781ade37" containerName="otc-container" Oct 30 11:16:17.889678 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:17.889618 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9907347d-d4b2-47e7-9754-5f9a781ade37" containerName="otc-container" Oct 30 11:16:17.889678 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:17.889674 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1ed3274-9c04-4de7-b397-a8d53269aee4" containerName="korrel8r" Oct 30 11:16:17.889839 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:17.889684 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="9907347d-d4b2-47e7-9754-5f9a781ade37" containerName="otc-container" Oct 30 11:16:17.893931 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:17.893914 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6" Oct 30 11:16:17.898388 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:17.898362 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:16:17.909185 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:17.909168 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 11:16:17.921195 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:17.921176 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 11:16:17.921324 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:17.921191 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 11:16:17.921324 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:17.921178 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 11:16:17.923505 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:17.923488 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-7qrq9\"" Oct 30 11:16:17.924819 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:17.924802 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6"] Oct 30 11:16:17.940046 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:17.940017 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c17e76ca-983d-4914-bffb-aaec3a52ea51-tls\") pod \"korrel8r-5d45699dc4-wc2h6\" (UID: \"c17e76ca-983d-4914-bffb-aaec3a52ea51\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6" Oct 30 11:16:17.940152 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:17.940065 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vv2n\" (UniqueName: \"kubernetes.io/projected/c17e76ca-983d-4914-bffb-aaec3a52ea51-kube-api-access-7vv2n\") pod \"korrel8r-5d45699dc4-wc2h6\" (UID: \"c17e76ca-983d-4914-bffb-aaec3a52ea51\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6" Oct 30 11:16:17.940195 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:17.940176 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c17e76ca-983d-4914-bffb-aaec3a52ea51-config\") pod \"korrel8r-5d45699dc4-wc2h6\" (UID: \"c17e76ca-983d-4914-bffb-aaec3a52ea51\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6" Oct 30 11:16:18.040812 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.040777 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c17e76ca-983d-4914-bffb-aaec3a52ea51-tls\") pod \"korrel8r-5d45699dc4-wc2h6\" (UID: \"c17e76ca-983d-4914-bffb-aaec3a52ea51\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6" Oct 30 11:16:18.040994 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.040819 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vv2n\" (UniqueName: \"kubernetes.io/projected/c17e76ca-983d-4914-bffb-aaec3a52ea51-kube-api-access-7vv2n\") pod \"korrel8r-5d45699dc4-wc2h6\" (UID: \"c17e76ca-983d-4914-bffb-aaec3a52ea51\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6" Oct 30 11:16:18.040994 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.040862 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c17e76ca-983d-4914-bffb-aaec3a52ea51-config\") pod \"korrel8r-5d45699dc4-wc2h6\" (UID: \"c17e76ca-983d-4914-bffb-aaec3a52ea51\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6" Oct 30 11:16:18.040994 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:16:18.040917 4289 secret.go:189] Couldn't get secret redhat-ods-monitoring/korrel8r-tls: secret "korrel8r-tls" not found Oct 30 11:16:18.040994 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:16:18.040991 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c17e76ca-983d-4914-bffb-aaec3a52ea51-tls podName:c17e76ca-983d-4914-bffb-aaec3a52ea51 nodeName:}" failed. No retries permitted until 2026-10-30 11:16:18.540974801 +0000 UTC m=+1869.042354265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/c17e76ca-983d-4914-bffb-aaec3a52ea51-tls") pod "korrel8r-5d45699dc4-wc2h6" (UID: "c17e76ca-983d-4914-bffb-aaec3a52ea51") : secret "korrel8r-tls" not found Oct 30 11:16:18.041525 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.041509 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c17e76ca-983d-4914-bffb-aaec3a52ea51-config\") pod \"korrel8r-5d45699dc4-wc2h6\" (UID: \"c17e76ca-983d-4914-bffb-aaec3a52ea51\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6" Oct 30 11:16:18.064020 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.063993 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vv2n\" (UniqueName: \"kubernetes.io/projected/c17e76ca-983d-4914-bffb-aaec3a52ea51-kube-api-access-7vv2n\") pod \"korrel8r-5d45699dc4-wc2h6\" (UID: \"c17e76ca-983d-4914-bffb-aaec3a52ea51\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6" Oct 30 11:16:18.076661 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.076633 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:16:18.243332 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.243295 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:16:18.544118 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.544024 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c17e76ca-983d-4914-bffb-aaec3a52ea51-tls\") pod \"korrel8r-5d45699dc4-wc2h6\" (UID: \"c17e76ca-983d-4914-bffb-aaec3a52ea51\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6" Oct 30 11:16:18.546562 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.546534 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c17e76ca-983d-4914-bffb-aaec3a52ea51-tls\") pod \"korrel8r-5d45699dc4-wc2h6\" (UID: \"c17e76ca-983d-4914-bffb-aaec3a52ea51\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6" Oct 30 11:16:18.614585 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.614554 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:16:18.617971 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.617941 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:16:18.644967 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.644942 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:16:18.654835 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.654811 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 11:16:18.654981 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.654941 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-789dw\"" Oct 30 11:16:18.677015 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.676982 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:16:18.682826 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.682801 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-0ed87044\"" Oct 30 11:16:18.746134 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.746099 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfc7m\" (UniqueName: \"kubernetes.io/projected/0cd57bf1-a894-4969-839b-c46e65e393f6-kube-api-access-vfc7m\") pod \"data-science-collector-collector-1\" (UID: \"0cd57bf1-a894-4969-839b-c46e65e393f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:16:18.746290 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.746142 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0cd57bf1-a894-4969-839b-c46e65e393f6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0cd57bf1-a894-4969-839b-c46e65e393f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:16:18.746290 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.746162 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd57bf1-a894-4969-839b-c46e65e393f6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0cd57bf1-a894-4969-839b-c46e65e393f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:16:18.806125 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.806046 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6" Oct 30 11:16:18.847322 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.847285 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfc7m\" (UniqueName: \"kubernetes.io/projected/0cd57bf1-a894-4969-839b-c46e65e393f6-kube-api-access-vfc7m\") pod \"data-science-collector-collector-1\" (UID: \"0cd57bf1-a894-4969-839b-c46e65e393f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:16:18.847505 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.847334 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0cd57bf1-a894-4969-839b-c46e65e393f6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0cd57bf1-a894-4969-839b-c46e65e393f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:16:18.847505 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.847465 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd57bf1-a894-4969-839b-c46e65e393f6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0cd57bf1-a894-4969-839b-c46e65e393f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:16:18.848025 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.848006 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0cd57bf1-a894-4969-839b-c46e65e393f6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0cd57bf1-a894-4969-839b-c46e65e393f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:16:18.850040 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.849992 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd57bf1-a894-4969-839b-c46e65e393f6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0cd57bf1-a894-4969-839b-c46e65e393f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:16:18.868574 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.868526 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:16:18.871305 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.871248 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfc7m\" (UniqueName: \"kubernetes.io/projected/0cd57bf1-a894-4969-839b-c46e65e393f6-kube-api-access-vfc7m\") pod \"data-science-collector-collector-1\" (UID: \"0cd57bf1-a894-4969-839b-c46e65e393f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:16:18.929816 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:18.929776 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:16:19.102664 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:19.102638 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6"] Oct 30 11:16:19.102899 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:16:19.102875 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17e76ca_983d_4914_bffb_aaec3a52ea51.slice/crio-fbc141b081eedb6d2674ef86a2886839c0b59e64b76d077da426b347d7b27dd1 WatchSource:0}: Error finding container fbc141b081eedb6d2674ef86a2886839c0b59e64b76d077da426b347d7b27dd1: Status 404 returned error can't find the container with id fbc141b081eedb6d2674ef86a2886839c0b59e64b76d077da426b347d7b27dd1 Oct 30 11:16:19.224204 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:19.224177 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:16:19.224346 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:16:19.224169 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd57bf1_a894_4969_839b_c46e65e393f6.slice/crio-64655a07589d6bc2d4f9a06468520c97554f12fc9dbe576f14f851315fbef33d WatchSource:0}: Error finding container 64655a07589d6bc2d4f9a06468520c97554f12fc9dbe576f14f851315fbef33d: Status 404 returned error can't find the container with id 64655a07589d6bc2d4f9a06468520c97554f12fc9dbe576f14f851315fbef33d Oct 30 11:16:19.226553 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:19.226520 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:16:19.226640 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:19.226582 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:16:19.745036 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:19.745003 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0cd57bf1-a894-4969-839b-c46e65e393f6","Type":"ContainerStarted","Data":"b180d2000ec3fe7d4f7cc98c5e8ba1d2ca5338d6172b01642624ff5d81a45988"} Oct 30 11:16:19.745230 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:19.745053 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0cd57bf1-a894-4969-839b-c46e65e393f6","Type":"ContainerStarted","Data":"64655a07589d6bc2d4f9a06468520c97554f12fc9dbe576f14f851315fbef33d"} Oct 30 11:16:19.746488 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:19.746462 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6" event={"ID":"c17e76ca-983d-4914-bffb-aaec3a52ea51","Type":"ContainerStarted","Data":"5418e53471feae52fa3ed419c04fceb1a71e1f2442f51082d3efcc06153a1808"} Oct 30 11:16:19.746593 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:19.746504 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6" event={"ID":"c17e76ca-983d-4914-bffb-aaec3a52ea51","Type":"ContainerStarted","Data":"fbc141b081eedb6d2674ef86a2886839c0b59e64b76d077da426b347d7b27dd1"} Oct 30 11:16:19.746593 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:19.746528 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6" Oct 30 11:16:19.786099 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:19.786053 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.78604087 podStartE2EDuration="1.78604087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:16:19.785375144 +0000 UTC m=+1870.286754629" watchObservedRunningTime="2026-10-30 11:16:19.78604087 +0000 UTC m=+1870.287420355" Oct 30 11:16:19.826586 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:19.826537 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6" podStartSLOduration=2.826520512 podStartE2EDuration="2.826520512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:16:19.825088859 +0000 UTC m=+1870.326468356" watchObservedRunningTime="2026-10-30 11:16:19.826520512 +0000 UTC m=+1870.327899991" Oct 30 11:16:22.799984 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:22.799931 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:16:22.979162 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:22.979114 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:16:23.169482 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.169374 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:16:23.262458 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.262411 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b"] Oct 30 11:16:23.265939 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.265921 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:23.282163 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.282136 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 30 11:16:23.282284 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.282207 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ca-bundle\"" Oct 30 11:16:23.293373 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.293345 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-cabundle\"" Oct 30 11:16:23.293558 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.293422 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-dockercfg-k4tkl\"" Oct 30 11:16:23.293558 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.293498 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 30 11:16:23.293705 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.293590 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-tls\"" Oct 30 11:16:23.293705 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.293660 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-mtls\"" Oct 30 11:16:23.295162 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.295143 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b"] Oct 30 11:16:23.337559 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.337530 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:16:23.386519 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.386492 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/27693017-f007-46a5-8f5c-b4ec73dcb984-serving-certs\") pod \"tempo-data-science-tempostack-gateway-78c5f7b646-rq99b\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:23.386682 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.386530 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42crt\" (UniqueName: \"kubernetes.io/projected/27693017-f007-46a5-8f5c-b4ec73dcb984-kube-api-access-42crt\") pod \"tempo-data-science-tempostack-gateway-78c5f7b646-rq99b\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:23.386747 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.386648 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/27693017-f007-46a5-8f5c-b4ec73dcb984-tenant\") pod \"tempo-data-science-tempostack-gateway-78c5f7b646-rq99b\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:23.386747 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.386736 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/27693017-f007-46a5-8f5c-b4ec73dcb984-rbac\") pod \"tempo-data-science-tempostack-gateway-78c5f7b646-rq99b\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:23.386894 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.386870 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/27693017-f007-46a5-8f5c-b4ec73dcb984-cabundle\") pod \"tempo-data-science-tempostack-gateway-78c5f7b646-rq99b\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:23.386941 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.386920 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/27693017-f007-46a5-8f5c-b4ec73dcb984-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-78c5f7b646-rq99b\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:23.387057 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.387036 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27693017-f007-46a5-8f5c-b4ec73dcb984-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-78c5f7b646-rq99b\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:23.488279 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.488246 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27693017-f007-46a5-8f5c-b4ec73dcb984-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-78c5f7b646-rq99b\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:23.488493 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.488288 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/27693017-f007-46a5-8f5c-b4ec73dcb984-serving-certs\") pod \"tempo-data-science-tempostack-gateway-78c5f7b646-rq99b\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:23.488493 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.488312 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42crt\" (UniqueName: \"kubernetes.io/projected/27693017-f007-46a5-8f5c-b4ec73dcb984-kube-api-access-42crt\") pod \"tempo-data-science-tempostack-gateway-78c5f7b646-rq99b\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:23.488493 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.488333 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/27693017-f007-46a5-8f5c-b4ec73dcb984-tenant\") pod \"tempo-data-science-tempostack-gateway-78c5f7b646-rq99b\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:23.488493 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.488371 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/27693017-f007-46a5-8f5c-b4ec73dcb984-rbac\") pod \"tempo-data-science-tempostack-gateway-78c5f7b646-rq99b\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:23.488493 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.488418 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/27693017-f007-46a5-8f5c-b4ec73dcb984-cabundle\") pod \"tempo-data-science-tempostack-gateway-78c5f7b646-rq99b\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:23.488493 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.488472 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/27693017-f007-46a5-8f5c-b4ec73dcb984-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-78c5f7b646-rq99b\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:23.489099 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.489070 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27693017-f007-46a5-8f5c-b4ec73dcb984-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-78c5f7b646-rq99b\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:23.489221 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.489084 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/27693017-f007-46a5-8f5c-b4ec73dcb984-rbac\") pod \"tempo-data-science-tempostack-gateway-78c5f7b646-rq99b\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:23.489221 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.489110 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/27693017-f007-46a5-8f5c-b4ec73dcb984-cabundle\") pod \"tempo-data-science-tempostack-gateway-78c5f7b646-rq99b\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:23.490934 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.490909 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/27693017-f007-46a5-8f5c-b4ec73dcb984-tenant\") pod \"tempo-data-science-tempostack-gateway-78c5f7b646-rq99b\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:23.491034 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.491017 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/27693017-f007-46a5-8f5c-b4ec73dcb984-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-78c5f7b646-rq99b\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:23.491079 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.491046 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/27693017-f007-46a5-8f5c-b4ec73dcb984-serving-certs\") pod \"tempo-data-science-tempostack-gateway-78c5f7b646-rq99b\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:23.515833 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.515803 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42crt\" (UniqueName: \"kubernetes.io/projected/27693017-f007-46a5-8f5c-b4ec73dcb984-kube-api-access-42crt\") pod \"tempo-data-science-tempostack-gateway-78c5f7b646-rq99b\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:23.580720 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.580688 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:23.914423 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:23.914392 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b"] Oct 30 11:16:23.915249 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:16:23.915227 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27693017_f007_46a5_8f5c_b4ec73dcb984.slice/crio-9b11062945923180fcbcd5b06cf3ad84c7ca50671ab6733f70dfa51916ee6225 WatchSource:0}: Error finding container 9b11062945923180fcbcd5b06cf3ad84c7ca50671ab6733f70dfa51916ee6225: Status 404 returned error can't find the container with id 9b11062945923180fcbcd5b06cf3ad84c7ca50671ab6733f70dfa51916ee6225 Oct 30 11:16:24.770414 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:24.770374 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" event={"ID":"27693017-f007-46a5-8f5c-b4ec73dcb984","Type":"ContainerStarted","Data":"dfaa0a981a2da1b53d7fb112ecf708acb3beed86ec2fb9a0266143672e8098f7"} Oct 30 11:16:24.770596 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:24.770456 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" event={"ID":"27693017-f007-46a5-8f5c-b4ec73dcb984","Type":"ContainerStarted","Data":"a865d17a7d4ad5c8b65ad14749686cdd8dcc2960b794be6959d68b5c26a47e76"} Oct 30 11:16:24.770596 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:24.770488 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" event={"ID":"27693017-f007-46a5-8f5c-b4ec73dcb984","Type":"ContainerStarted","Data":"9b11062945923180fcbcd5b06cf3ad84c7ca50671ab6733f70dfa51916ee6225"} Oct 30 11:16:24.770702 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:24.770606 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:24.772034 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:24.772007 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" podUID="27693017-f007-46a5-8f5c-b4ec73dcb984" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.134.0.66:8083/ready\": dial tcp 10.134.0.66:8083: connect: connection refused" Oct 30 11:16:24.838818 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:24.838760 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" podStartSLOduration=1.8387464 podStartE2EDuration="1.8387464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:16:24.826716385 +0000 UTC m=+1875.328095908" watchObservedRunningTime="2026-10-30 11:16:24.8387464 +0000 UTC m=+1875.340125884" Oct 30 11:16:25.781355 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:25.781320 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:25.781852 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:25.781565 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" podUID="27693017-f007-46a5-8f5c-b4ec73dcb984" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.134.0.66:8083/ready\": dial tcp 10.134.0.66:8083: connect: connection refused" Oct 30 11:16:25.978736 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:25.978707 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:26.257461 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:16:26.257401 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:16:26.787573 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:26.787547 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:26.859882 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:26.859849 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b"] Oct 30 11:16:28.795670 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:28.795610 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" podUID="27693017-f007-46a5-8f5c-b4ec73dcb984" containerName="tempo-gateway" containerID="cri-o://a865d17a7d4ad5c8b65ad14749686cdd8dcc2960b794be6959d68b5c26a47e76" gracePeriod=30 Oct 30 11:16:28.796206 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:28.795644 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" podUID="27693017-f007-46a5-8f5c-b4ec73dcb984" containerName="tempo-gateway-opa" containerID="cri-o://dfaa0a981a2da1b53d7fb112ecf708acb3beed86ec2fb9a0266143672e8098f7" gracePeriod=30 Oct 30 11:16:29.801476 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:29.801423 4289 generic.go:356] "Generic (PLEG): container finished" podID="27693017-f007-46a5-8f5c-b4ec73dcb984" containerID="dfaa0a981a2da1b53d7fb112ecf708acb3beed86ec2fb9a0266143672e8098f7" exitCode=0 Oct 30 11:16:29.801873 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:29.801495 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" event={"ID":"27693017-f007-46a5-8f5c-b4ec73dcb984","Type":"ContainerDied","Data":"dfaa0a981a2da1b53d7fb112ecf708acb3beed86ec2fb9a0266143672e8098f7"} Oct 30 11:16:30.040206 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.040172 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:30.148111 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.148031 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27693017-f007-46a5-8f5c-b4ec73dcb984-tenant\" (UniqueName: \"kubernetes.io/secret/27693017-f007-46a5-8f5c-b4ec73dcb984-tenant\") pod \"27693017-f007-46a5-8f5c-b4ec73dcb984\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " Oct 30 11:16:30.148111 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.148084 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27693017-f007-46a5-8f5c-b4ec73dcb984-tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/27693017-f007-46a5-8f5c-b4ec73dcb984-tempo-data-science-tempostack-gateway-mtls\") pod \"27693017-f007-46a5-8f5c-b4ec73dcb984\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " Oct 30 11:16:30.148111 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.148108 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27693017-f007-46a5-8f5c-b4ec73dcb984-serving-certs\" (UniqueName: \"kubernetes.io/secret/27693017-f007-46a5-8f5c-b4ec73dcb984-serving-certs\") pod \"27693017-f007-46a5-8f5c-b4ec73dcb984\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " Oct 30 11:16:30.148376 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.148134 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27693017-f007-46a5-8f5c-b4ec73dcb984-kube-api-access-42crt\" (UniqueName: \"kubernetes.io/projected/27693017-f007-46a5-8f5c-b4ec73dcb984-kube-api-access-42crt\") pod \"27693017-f007-46a5-8f5c-b4ec73dcb984\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " Oct 30 11:16:30.148376 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.148158 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27693017-f007-46a5-8f5c-b4ec73dcb984-cabundle\" (UniqueName: \"kubernetes.io/configmap/27693017-f007-46a5-8f5c-b4ec73dcb984-cabundle\") pod \"27693017-f007-46a5-8f5c-b4ec73dcb984\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " Oct 30 11:16:30.148376 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.148298 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27693017-f007-46a5-8f5c-b4ec73dcb984-rbac\" (UniqueName: \"kubernetes.io/configmap/27693017-f007-46a5-8f5c-b4ec73dcb984-rbac\") pod \"27693017-f007-46a5-8f5c-b4ec73dcb984\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " Oct 30 11:16:30.148376 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.148326 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27693017-f007-46a5-8f5c-b4ec73dcb984-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27693017-f007-46a5-8f5c-b4ec73dcb984-tempo-data-science-tempostack-ca-bundle\") pod \"27693017-f007-46a5-8f5c-b4ec73dcb984\" (UID: \"27693017-f007-46a5-8f5c-b4ec73dcb984\") " Oct 30 11:16:30.152773 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.152741 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27693017-f007-46a5-8f5c-b4ec73dcb984-cabundle" pod "27693017-f007-46a5-8f5c-b4ec73dcb984" (UID: "27693017-f007-46a5-8f5c-b4ec73dcb984"). InnerVolumeSpecName "cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:16:30.152773 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.152764 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27693017-f007-46a5-8f5c-b4ec73dcb984-tempo-data-science-tempostack-ca-bundle" pod "27693017-f007-46a5-8f5c-b4ec73dcb984" (UID: "27693017-f007-46a5-8f5c-b4ec73dcb984"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:16:30.152977 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.152936 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27693017-f007-46a5-8f5c-b4ec73dcb984-rbac" pod "27693017-f007-46a5-8f5c-b4ec73dcb984" (UID: "27693017-f007-46a5-8f5c-b4ec73dcb984"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:16:30.154373 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.154344 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27693017-f007-46a5-8f5c-b4ec73dcb984-serving-certs" pod "27693017-f007-46a5-8f5c-b4ec73dcb984" (UID: "27693017-f007-46a5-8f5c-b4ec73dcb984"). InnerVolumeSpecName "serving-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:16:30.154568 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.154523 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27693017-f007-46a5-8f5c-b4ec73dcb984-tempo-data-science-tempostack-gateway-mtls" pod "27693017-f007-46a5-8f5c-b4ec73dcb984" (UID: "27693017-f007-46a5-8f5c-b4ec73dcb984"). InnerVolumeSpecName "tempo-data-science-tempostack-gateway-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:16:30.154713 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.154694 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27693017-f007-46a5-8f5c-b4ec73dcb984-tenant" pod "27693017-f007-46a5-8f5c-b4ec73dcb984" (UID: "27693017-f007-46a5-8f5c-b4ec73dcb984"). InnerVolumeSpecName "tenant". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:16:30.154815 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.154795 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27693017-f007-46a5-8f5c-b4ec73dcb984-kube-api-access-42crt" pod "27693017-f007-46a5-8f5c-b4ec73dcb984" (UID: "27693017-f007-46a5-8f5c-b4ec73dcb984"). InnerVolumeSpecName "kube-api-access-42crt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:16:30.249410 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.249373 4289 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/27693017-f007-46a5-8f5c-b4ec73dcb984-tempo-data-science-tempostack-gateway-mtls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:16:30.249410 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.249407 4289 reconciler_common.go:299] "Volume detached for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/27693017-f007-46a5-8f5c-b4ec73dcb984-serving-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:16:30.249651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.249422 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42crt\" (UniqueName: \"kubernetes.io/projected/27693017-f007-46a5-8f5c-b4ec73dcb984-kube-api-access-42crt\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:16:30.249651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.249458 4289 reconciler_common.go:299] "Volume detached for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/27693017-f007-46a5-8f5c-b4ec73dcb984-cabundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:16:30.249651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.249471 4289 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/27693017-f007-46a5-8f5c-b4ec73dcb984-rbac\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:16:30.249651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.249482 4289 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27693017-f007-46a5-8f5c-b4ec73dcb984-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:16:30.249651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.249492 4289 reconciler_common.go:299] "Volume detached for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/27693017-f007-46a5-8f5c-b4ec73dcb984-tenant\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:16:30.752934 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.752899 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6" Oct 30 11:16:30.807944 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.807911 4289 generic.go:356] "Generic (PLEG): container finished" podID="27693017-f007-46a5-8f5c-b4ec73dcb984" containerID="a865d17a7d4ad5c8b65ad14749686cdd8dcc2960b794be6959d68b5c26a47e76" exitCode=0 Oct 30 11:16:30.808356 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.807993 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" Oct 30 11:16:30.808356 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.807998 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" event={"ID":"27693017-f007-46a5-8f5c-b4ec73dcb984","Type":"ContainerDied","Data":"a865d17a7d4ad5c8b65ad14749686cdd8dcc2960b794be6959d68b5c26a47e76"} Oct 30 11:16:30.808356 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.808057 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b" event={"ID":"27693017-f007-46a5-8f5c-b4ec73dcb984","Type":"ContainerDied","Data":"9b11062945923180fcbcd5b06cf3ad84c7ca50671ab6733f70dfa51916ee6225"} Oct 30 11:16:30.808356 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.808084 4289 scope.go:122] "RemoveContainer" containerID="dfaa0a981a2da1b53d7fb112ecf708acb3beed86ec2fb9a0266143672e8098f7" Oct 30 11:16:30.817366 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.817347 4289 scope.go:122] "RemoveContainer" containerID="a865d17a7d4ad5c8b65ad14749686cdd8dcc2960b794be6959d68b5c26a47e76" Oct 30 11:16:30.827016 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.826997 4289 scope.go:122] "RemoveContainer" containerID="dfaa0a981a2da1b53d7fb112ecf708acb3beed86ec2fb9a0266143672e8098f7" Oct 30 11:16:30.827312 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:16:30.827291 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfaa0a981a2da1b53d7fb112ecf708acb3beed86ec2fb9a0266143672e8098f7\": container with ID starting with dfaa0a981a2da1b53d7fb112ecf708acb3beed86ec2fb9a0266143672e8098f7 not found: ID does not exist" containerID="dfaa0a981a2da1b53d7fb112ecf708acb3beed86ec2fb9a0266143672e8098f7" Oct 30 11:16:30.827393 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.827318 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfaa0a981a2da1b53d7fb112ecf708acb3beed86ec2fb9a0266143672e8098f7"} err="failed to get container status \"dfaa0a981a2da1b53d7fb112ecf708acb3beed86ec2fb9a0266143672e8098f7\": rpc error: code = NotFound desc = could not find container \"dfaa0a981a2da1b53d7fb112ecf708acb3beed86ec2fb9a0266143672e8098f7\": container with ID starting with dfaa0a981a2da1b53d7fb112ecf708acb3beed86ec2fb9a0266143672e8098f7 not found: ID does not exist" Oct 30 11:16:30.827393 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.827334 4289 scope.go:122] "RemoveContainer" containerID="a865d17a7d4ad5c8b65ad14749686cdd8dcc2960b794be6959d68b5c26a47e76" Oct 30 11:16:30.827635 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:16:30.827619 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a865d17a7d4ad5c8b65ad14749686cdd8dcc2960b794be6959d68b5c26a47e76\": container with ID starting with a865d17a7d4ad5c8b65ad14749686cdd8dcc2960b794be6959d68b5c26a47e76 not found: ID does not exist" containerID="a865d17a7d4ad5c8b65ad14749686cdd8dcc2960b794be6959d68b5c26a47e76" Oct 30 11:16:30.827703 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.827639 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a865d17a7d4ad5c8b65ad14749686cdd8dcc2960b794be6959d68b5c26a47e76"} err="failed to get container status \"a865d17a7d4ad5c8b65ad14749686cdd8dcc2960b794be6959d68b5c26a47e76\": rpc error: code = NotFound desc = could not find container \"a865d17a7d4ad5c8b65ad14749686cdd8dcc2960b794be6959d68b5c26a47e76\": container with ID starting with a865d17a7d4ad5c8b65ad14749686cdd8dcc2960b794be6959d68b5c26a47e76 not found: ID does not exist" Oct 30 11:16:30.858928 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.858903 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b"] Oct 30 11:16:30.871102 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:30.871076 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-78c5f7b646-rq99b"] Oct 30 11:16:31.624770 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:31.624741 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27693017-f007-46a5-8f5c-b4ec73dcb984" path="/var/lib/kubelet/pods/27693017-f007-46a5-8f5c-b4ec73dcb984/volumes" Oct 30 11:16:40.850032 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:40.849991 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:16:44.999961 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:44.999927 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:16:56.257706 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:16:56.257661 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:16:57.074404 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:16:57.074368 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:17:18.574731 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:18.574687 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:17:18.744670 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:18.744633 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:17:26.257392 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:17:26.257310 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:17:32.791179 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:32.791141 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:17:32.987055 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:32.987015 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:17:33.308470 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:33.308415 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:17:33.308669 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:33.308647 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="0cd57bf1-a894-4969-839b-c46e65e393f6" containerName="otc-container" containerID="cri-o://b180d2000ec3fe7d4f7cc98c5e8ba1d2ca5338d6172b01642624ff5d81a45988" gracePeriod=30 Oct 30 11:17:33.488990 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:33.488956 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:17:33.550214 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:33.550190 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:17:33.661980 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:33.661901 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:17:33.671507 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:33.671485 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cd57bf1-a894-4969-839b-c46e65e393f6-otc-internal\" (UniqueName: \"kubernetes.io/configmap/0cd57bf1-a894-4969-839b-c46e65e393f6-otc-internal\") pod \"0cd57bf1-a894-4969-839b-c46e65e393f6\" (UID: \"0cd57bf1-a894-4969-839b-c46e65e393f6\") " Oct 30 11:17:33.671613 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:33.671529 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cd57bf1-a894-4969-839b-c46e65e393f6-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd57bf1-a894-4969-839b-c46e65e393f6-monitoring-tls-certs\") pod \"0cd57bf1-a894-4969-839b-c46e65e393f6\" (UID: \"0cd57bf1-a894-4969-839b-c46e65e393f6\") " Oct 30 11:17:33.671613 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:33.671559 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd57bf1-a894-4969-839b-c46e65e393f6-kube-api-access-vfc7m\" (UniqueName: \"kubernetes.io/projected/0cd57bf1-a894-4969-839b-c46e65e393f6-kube-api-access-vfc7m\") pod \"0cd57bf1-a894-4969-839b-c46e65e393f6\" (UID: \"0cd57bf1-a894-4969-839b-c46e65e393f6\") " Oct 30 11:17:33.671865 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:33.671817 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd57bf1-a894-4969-839b-c46e65e393f6-otc-internal" pod "0cd57bf1-a894-4969-839b-c46e65e393f6" (UID: "0cd57bf1-a894-4969-839b-c46e65e393f6"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:17:33.673675 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:33.673646 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cd57bf1-a894-4969-839b-c46e65e393f6-monitoring-tls-certs" pod "0cd57bf1-a894-4969-839b-c46e65e393f6" (UID: "0cd57bf1-a894-4969-839b-c46e65e393f6"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:17:33.673761 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:33.673727 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd57bf1-a894-4969-839b-c46e65e393f6-kube-api-access-vfc7m" pod "0cd57bf1-a894-4969-839b-c46e65e393f6" (UID: "0cd57bf1-a894-4969-839b-c46e65e393f6"). InnerVolumeSpecName "kube-api-access-vfc7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:17:33.772896 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:33.772865 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0cd57bf1-a894-4969-839b-c46e65e393f6-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:17:33.772896 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:33.772896 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd57bf1-a894-4969-839b-c46e65e393f6-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:17:33.773092 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:33.772910 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfc7m\" (UniqueName: \"kubernetes.io/projected/0cd57bf1-a894-4969-839b-c46e65e393f6-kube-api-access-vfc7m\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:17:34.070578 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.070545 4289 generic.go:356] "Generic (PLEG): container finished" podID="0cd57bf1-a894-4969-839b-c46e65e393f6" containerID="b180d2000ec3fe7d4f7cc98c5e8ba1d2ca5338d6172b01642624ff5d81a45988" exitCode=0 Oct 30 11:17:34.071081 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.070604 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:17:34.071081 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.070635 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0cd57bf1-a894-4969-839b-c46e65e393f6","Type":"ContainerDied","Data":"b180d2000ec3fe7d4f7cc98c5e8ba1d2ca5338d6172b01642624ff5d81a45988"} Oct 30 11:17:34.071081 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.070689 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0cd57bf1-a894-4969-839b-c46e65e393f6","Type":"ContainerDied","Data":"64655a07589d6bc2d4f9a06468520c97554f12fc9dbe576f14f851315fbef33d"} Oct 30 11:17:34.071081 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.070711 4289 scope.go:122] "RemoveContainer" containerID="b180d2000ec3fe7d4f7cc98c5e8ba1d2ca5338d6172b01642624ff5d81a45988" Oct 30 11:17:34.079316 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.079300 4289 scope.go:122] "RemoveContainer" containerID="b180d2000ec3fe7d4f7cc98c5e8ba1d2ca5338d6172b01642624ff5d81a45988" Oct 30 11:17:34.079606 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:17:34.079589 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b180d2000ec3fe7d4f7cc98c5e8ba1d2ca5338d6172b01642624ff5d81a45988\": container with ID starting with b180d2000ec3fe7d4f7cc98c5e8ba1d2ca5338d6172b01642624ff5d81a45988 not found: ID does not exist" containerID="b180d2000ec3fe7d4f7cc98c5e8ba1d2ca5338d6172b01642624ff5d81a45988" Oct 30 11:17:34.079674 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.079615 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b180d2000ec3fe7d4f7cc98c5e8ba1d2ca5338d6172b01642624ff5d81a45988"} err="failed to get container status \"b180d2000ec3fe7d4f7cc98c5e8ba1d2ca5338d6172b01642624ff5d81a45988\": rpc error: code = NotFound desc = could not find container \"b180d2000ec3fe7d4f7cc98c5e8ba1d2ca5338d6172b01642624ff5d81a45988\": container with ID starting with b180d2000ec3fe7d4f7cc98c5e8ba1d2ca5338d6172b01642624ff5d81a45988 not found: ID does not exist" Oct 30 11:17:34.121784 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.121756 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:17:34.133713 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.133688 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:17:34.167487 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.167456 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:17:34.167784 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.167772 4289 cpu_manager.go:414] "containerMap: removing container" podUID="27693017-f007-46a5-8f5c-b4ec73dcb984" containerName="tempo-gateway" Oct 30 11:17:34.167835 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.167786 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27693017-f007-46a5-8f5c-b4ec73dcb984" containerName="tempo-gateway" Oct 30 11:17:34.167835 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.167804 4289 cpu_manager.go:414] "containerMap: removing container" podUID="0cd57bf1-a894-4969-839b-c46e65e393f6" containerName="otc-container" Oct 30 11:17:34.167835 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.167811 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd57bf1-a894-4969-839b-c46e65e393f6" containerName="otc-container" Oct 30 11:17:34.167929 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.167836 4289 cpu_manager.go:414] "containerMap: removing container" podUID="27693017-f007-46a5-8f5c-b4ec73dcb984" containerName="tempo-gateway-opa" Oct 30 11:17:34.167929 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.167842 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27693017-f007-46a5-8f5c-b4ec73dcb984" containerName="tempo-gateway-opa" Oct 30 11:17:34.167929 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.167904 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd57bf1-a894-4969-839b-c46e65e393f6" containerName="otc-container" Oct 30 11:17:34.167929 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.167915 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="27693017-f007-46a5-8f5c-b4ec73dcb984" containerName="tempo-gateway-opa" Oct 30 11:17:34.167929 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.167924 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="27693017-f007-46a5-8f5c-b4ec73dcb984" containerName="tempo-gateway" Oct 30 11:17:34.172213 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.172181 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:17:34.189373 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.189337 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-789dw\"" Oct 30 11:17:34.189546 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.189499 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-0ed87044\"" Oct 30 11:17:34.189616 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.189559 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 11:17:34.191112 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.191090 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:17:34.276514 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.276475 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab9ecb55-83a5-4c09-a623-68419ffbf1c0-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ab9ecb55-83a5-4c09-a623-68419ffbf1c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:17:34.276514 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.276515 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab9ecb55-83a5-4c09-a623-68419ffbf1c0-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ab9ecb55-83a5-4c09-a623-68419ffbf1c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:17:34.276723 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.276651 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l25vn\" (UniqueName: \"kubernetes.io/projected/ab9ecb55-83a5-4c09-a623-68419ffbf1c0-kube-api-access-l25vn\") pod \"data-science-collector-collector-1\" (UID: \"ab9ecb55-83a5-4c09-a623-68419ffbf1c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:17:34.377754 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.377671 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l25vn\" (UniqueName: \"kubernetes.io/projected/ab9ecb55-83a5-4c09-a623-68419ffbf1c0-kube-api-access-l25vn\") pod \"data-science-collector-collector-1\" (UID: \"ab9ecb55-83a5-4c09-a623-68419ffbf1c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:17:34.377754 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.377721 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab9ecb55-83a5-4c09-a623-68419ffbf1c0-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ab9ecb55-83a5-4c09-a623-68419ffbf1c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:17:34.377947 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.377756 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab9ecb55-83a5-4c09-a623-68419ffbf1c0-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ab9ecb55-83a5-4c09-a623-68419ffbf1c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:17:34.378574 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.378543 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab9ecb55-83a5-4c09-a623-68419ffbf1c0-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ab9ecb55-83a5-4c09-a623-68419ffbf1c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:17:34.380221 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.380196 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab9ecb55-83a5-4c09-a623-68419ffbf1c0-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ab9ecb55-83a5-4c09-a623-68419ffbf1c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:17:34.400551 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.400528 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l25vn\" (UniqueName: \"kubernetes.io/projected/ab9ecb55-83a5-4c09-a623-68419ffbf1c0-kube-api-access-l25vn\") pod \"data-science-collector-collector-1\" (UID: \"ab9ecb55-83a5-4c09-a623-68419ffbf1c0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:17:34.484600 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.484566 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:17:34.778309 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.778284 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:17:34.778825 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:17:34.778802 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab9ecb55_83a5_4c09_a623_68419ffbf1c0.slice/crio-a0c413298aff3be52dd2edc50a31efe079f44cecc52d3debde6ab6f00b76befe WatchSource:0}: Error finding container a0c413298aff3be52dd2edc50a31efe079f44cecc52d3debde6ab6f00b76befe: Status 404 returned error can't find the container with id a0c413298aff3be52dd2edc50a31efe079f44cecc52d3debde6ab6f00b76befe Oct 30 11:17:34.781396 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.781363 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:17:34.781492 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:34.781419 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:17:35.076482 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:35.076374 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ab9ecb55-83a5-4c09-a623-68419ffbf1c0","Type":"ContainerStarted","Data":"85ac47d14cb10cfdad74cab643683c87c9d12be2a61cf613ca6702affbb3908f"} Oct 30 11:17:35.076482 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:35.076425 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ab9ecb55-83a5-4c09-a623-68419ffbf1c0","Type":"ContainerStarted","Data":"a0c413298aff3be52dd2edc50a31efe079f44cecc52d3debde6ab6f00b76befe"} Oct 30 11:17:35.121103 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:35.121046 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.121031786 podStartE2EDuration="1.121031786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:17:35.11950717 +0000 UTC m=+1945.620886655" watchObservedRunningTime="2026-10-30 11:17:35.121031786 +0000 UTC m=+1945.622411270" Oct 30 11:17:35.625243 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:35.625212 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cd57bf1-a894-4969-839b-c46e65e393f6" path="/var/lib/kubelet/pods/0cd57bf1-a894-4969-839b-c46e65e393f6/volumes" Oct 30 11:17:40.517320 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:40.517281 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:17:56.257926 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:17:56.257889 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:17:57.197937 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:17:57.197900 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:06.408037 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:06.407998 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6"] Oct 30 11:18:06.408604 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:06.408227 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6" podUID="c17e76ca-983d-4914-bffb-aaec3a52ea51" containerName="korrel8r" containerID="cri-o://5418e53471feae52fa3ed419c04fceb1a71e1f2442f51082d3efcc06153a1808" gracePeriod=30 Oct 30 11:18:06.669324 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:06.669263 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6" Oct 30 11:18:06.721768 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:06.721742 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c17e76ca-983d-4914-bffb-aaec3a52ea51-kube-api-access-7vv2n\" (UniqueName: \"kubernetes.io/projected/c17e76ca-983d-4914-bffb-aaec3a52ea51-kube-api-access-7vv2n\") pod \"c17e76ca-983d-4914-bffb-aaec3a52ea51\" (UID: \"c17e76ca-983d-4914-bffb-aaec3a52ea51\") " Oct 30 11:18:06.723962 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:06.723928 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c17e76ca-983d-4914-bffb-aaec3a52ea51-kube-api-access-7vv2n" pod "c17e76ca-983d-4914-bffb-aaec3a52ea51" (UID: "c17e76ca-983d-4914-bffb-aaec3a52ea51"). InnerVolumeSpecName "kube-api-access-7vv2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:18:06.777674 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:06.777641 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:18:06.777883 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:06.777857 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="ab9ecb55-83a5-4c09-a623-68419ffbf1c0" containerName="otc-container" containerID="cri-o://85ac47d14cb10cfdad74cab643683c87c9d12be2a61cf613ca6702affbb3908f" gracePeriod=30 Oct 30 11:18:06.822207 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:06.822172 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c17e76ca-983d-4914-bffb-aaec3a52ea51-tls\" (UniqueName: \"kubernetes.io/secret/c17e76ca-983d-4914-bffb-aaec3a52ea51-tls\") pod \"c17e76ca-983d-4914-bffb-aaec3a52ea51\" (UID: \"c17e76ca-983d-4914-bffb-aaec3a52ea51\") " Oct 30 11:18:06.822380 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:06.822214 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c17e76ca-983d-4914-bffb-aaec3a52ea51-config\" (UniqueName: \"kubernetes.io/configmap/c17e76ca-983d-4914-bffb-aaec3a52ea51-config\") pod \"c17e76ca-983d-4914-bffb-aaec3a52ea51\" (UID: \"c17e76ca-983d-4914-bffb-aaec3a52ea51\") " Oct 30 11:18:06.822380 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:06.822360 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vv2n\" (UniqueName: \"kubernetes.io/projected/c17e76ca-983d-4914-bffb-aaec3a52ea51-kube-api-access-7vv2n\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:18:06.822697 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:06.822672 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c17e76ca-983d-4914-bffb-aaec3a52ea51-config" pod "c17e76ca-983d-4914-bffb-aaec3a52ea51" (UID: "c17e76ca-983d-4914-bffb-aaec3a52ea51"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:18:06.825157 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:06.825131 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c17e76ca-983d-4914-bffb-aaec3a52ea51-tls" pod "c17e76ca-983d-4914-bffb-aaec3a52ea51" (UID: "c17e76ca-983d-4914-bffb-aaec3a52ea51"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:18:06.878719 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:06.878686 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:06.923832 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:06.923745 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c17e76ca-983d-4914-bffb-aaec3a52ea51-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:18:06.923832 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:06.923778 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c17e76ca-983d-4914-bffb-aaec3a52ea51-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:18:07.020784 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.020758 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:18:07.024827 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.024805 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab9ecb55-83a5-4c09-a623-68419ffbf1c0-kube-api-access-l25vn\" (UniqueName: \"kubernetes.io/projected/ab9ecb55-83a5-4c09-a623-68419ffbf1c0-kube-api-access-l25vn\") pod \"ab9ecb55-83a5-4c09-a623-68419ffbf1c0\" (UID: \"ab9ecb55-83a5-4c09-a623-68419ffbf1c0\") " Oct 30 11:18:07.024939 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.024846 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab9ecb55-83a5-4c09-a623-68419ffbf1c0-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab9ecb55-83a5-4c09-a623-68419ffbf1c0-otc-internal\") pod \"ab9ecb55-83a5-4c09-a623-68419ffbf1c0\" (UID: \"ab9ecb55-83a5-4c09-a623-68419ffbf1c0\") " Oct 30 11:18:07.025207 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.025184 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab9ecb55-83a5-4c09-a623-68419ffbf1c0-otc-internal" pod "ab9ecb55-83a5-4c09-a623-68419ffbf1c0" (UID: "ab9ecb55-83a5-4c09-a623-68419ffbf1c0"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:18:07.026973 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.026951 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab9ecb55-83a5-4c09-a623-68419ffbf1c0-kube-api-access-l25vn" pod "ab9ecb55-83a5-4c09-a623-68419ffbf1c0" (UID: "ab9ecb55-83a5-4c09-a623-68419ffbf1c0"). InnerVolumeSpecName "kube-api-access-l25vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:18:07.076295 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.076265 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:07.125974 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.125942 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab9ecb55-83a5-4c09-a623-68419ffbf1c0-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab9ecb55-83a5-4c09-a623-68419ffbf1c0-monitoring-tls-certs\") pod \"ab9ecb55-83a5-4c09-a623-68419ffbf1c0\" (UID: \"ab9ecb55-83a5-4c09-a623-68419ffbf1c0\") " Oct 30 11:18:07.126135 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.126090 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab9ecb55-83a5-4c09-a623-68419ffbf1c0-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:18:07.126135 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.126102 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l25vn\" (UniqueName: \"kubernetes.io/projected/ab9ecb55-83a5-4c09-a623-68419ffbf1c0-kube-api-access-l25vn\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:18:07.127866 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.127844 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab9ecb55-83a5-4c09-a623-68419ffbf1c0-monitoring-tls-certs" pod "ab9ecb55-83a5-4c09-a623-68419ffbf1c0" (UID: "ab9ecb55-83a5-4c09-a623-68419ffbf1c0"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:18:07.208830 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.208749 4289 generic.go:356] "Generic (PLEG): container finished" podID="c17e76ca-983d-4914-bffb-aaec3a52ea51" containerID="5418e53471feae52fa3ed419c04fceb1a71e1f2442f51082d3efcc06153a1808" exitCode=0 Oct 30 11:18:07.208830 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.208808 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6" Oct 30 11:18:07.209046 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.208840 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6" event={"ID":"c17e76ca-983d-4914-bffb-aaec3a52ea51","Type":"ContainerDied","Data":"5418e53471feae52fa3ed419c04fceb1a71e1f2442f51082d3efcc06153a1808"} Oct 30 11:18:07.209046 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.208903 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6" event={"ID":"c17e76ca-983d-4914-bffb-aaec3a52ea51","Type":"ContainerDied","Data":"fbc141b081eedb6d2674ef86a2886839c0b59e64b76d077da426b347d7b27dd1"} Oct 30 11:18:07.209046 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.208935 4289 scope.go:122] "RemoveContainer" containerID="5418e53471feae52fa3ed419c04fceb1a71e1f2442f51082d3efcc06153a1808" Oct 30 11:18:07.210368 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.210344 4289 generic.go:356] "Generic (PLEG): container finished" podID="ab9ecb55-83a5-4c09-a623-68419ffbf1c0" containerID="85ac47d14cb10cfdad74cab643683c87c9d12be2a61cf613ca6702affbb3908f" exitCode=0 Oct 30 11:18:07.210507 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.210385 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ab9ecb55-83a5-4c09-a623-68419ffbf1c0","Type":"ContainerDied","Data":"85ac47d14cb10cfdad74cab643683c87c9d12be2a61cf613ca6702affbb3908f"} Oct 30 11:18:07.210507 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.210405 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:18:07.210507 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.210423 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ab9ecb55-83a5-4c09-a623-68419ffbf1c0","Type":"ContainerDied","Data":"a0c413298aff3be52dd2edc50a31efe079f44cecc52d3debde6ab6f00b76befe"} Oct 30 11:18:07.218738 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.218723 4289 scope.go:122] "RemoveContainer" containerID="5418e53471feae52fa3ed419c04fceb1a71e1f2442f51082d3efcc06153a1808" Oct 30 11:18:07.219029 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:18:07.219007 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5418e53471feae52fa3ed419c04fceb1a71e1f2442f51082d3efcc06153a1808\": container with ID starting with 5418e53471feae52fa3ed419c04fceb1a71e1f2442f51082d3efcc06153a1808 not found: ID does not exist" containerID="5418e53471feae52fa3ed419c04fceb1a71e1f2442f51082d3efcc06153a1808" Oct 30 11:18:07.219085 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.219036 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5418e53471feae52fa3ed419c04fceb1a71e1f2442f51082d3efcc06153a1808"} err="failed to get container status \"5418e53471feae52fa3ed419c04fceb1a71e1f2442f51082d3efcc06153a1808\": rpc error: code = NotFound desc = could not find container \"5418e53471feae52fa3ed419c04fceb1a71e1f2442f51082d3efcc06153a1808\": container with ID starting with 5418e53471feae52fa3ed419c04fceb1a71e1f2442f51082d3efcc06153a1808 not found: ID does not exist" Oct 30 11:18:07.219085 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.219053 4289 scope.go:122] "RemoveContainer" containerID="85ac47d14cb10cfdad74cab643683c87c9d12be2a61cf613ca6702affbb3908f" Oct 30 11:18:07.226534 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.226415 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab9ecb55-83a5-4c09-a623-68419ffbf1c0-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:18:07.226640 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.226624 4289 scope.go:122] "RemoveContainer" containerID="85ac47d14cb10cfdad74cab643683c87c9d12be2a61cf613ca6702affbb3908f" Oct 30 11:18:07.226847 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:18:07.226831 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85ac47d14cb10cfdad74cab643683c87c9d12be2a61cf613ca6702affbb3908f\": container with ID starting with 85ac47d14cb10cfdad74cab643683c87c9d12be2a61cf613ca6702affbb3908f not found: ID does not exist" containerID="85ac47d14cb10cfdad74cab643683c87c9d12be2a61cf613ca6702affbb3908f" Oct 30 11:18:07.226895 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.226852 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85ac47d14cb10cfdad74cab643683c87c9d12be2a61cf613ca6702affbb3908f"} err="failed to get container status \"85ac47d14cb10cfdad74cab643683c87c9d12be2a61cf613ca6702affbb3908f\": rpc error: code = NotFound desc = could not find container \"85ac47d14cb10cfdad74cab643683c87c9d12be2a61cf613ca6702affbb3908f\": container with ID starting with 85ac47d14cb10cfdad74cab643683c87c9d12be2a61cf613ca6702affbb3908f not found: ID does not exist" Oct 30 11:18:07.250415 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.250392 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:07.260229 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.260208 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:18:07.266914 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.266893 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:18:07.300005 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.299980 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6"] Oct 30 11:18:07.302842 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.302821 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-wc2h6"] Oct 30 11:18:07.437913 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.437881 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:07.618944 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.618908 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:07.623592 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.623568 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab9ecb55-83a5-4c09-a623-68419ffbf1c0" path="/var/lib/kubelet/pods/ab9ecb55-83a5-4c09-a623-68419ffbf1c0/volumes" Oct 30 11:18:07.623961 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:07.623948 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c17e76ca-983d-4914-bffb-aaec3a52ea51" path="/var/lib/kubelet/pods/c17e76ca-983d-4914-bffb-aaec3a52ea51/volumes" Oct 30 11:18:10.216249 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:10.216163 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:11.263957 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.263923 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:11.269221 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.269196 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-lx4r9"] Oct 30 11:18:11.269518 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.269505 4289 cpu_manager.go:414] "containerMap: removing container" podUID="ab9ecb55-83a5-4c09-a623-68419ffbf1c0" containerName="otc-container" Oct 30 11:18:11.269563 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.269519 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab9ecb55-83a5-4c09-a623-68419ffbf1c0" containerName="otc-container" Oct 30 11:18:11.269563 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.269547 4289 cpu_manager.go:414] "containerMap: removing container" podUID="c17e76ca-983d-4914-bffb-aaec3a52ea51" containerName="korrel8r" Oct 30 11:18:11.269563 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.269552 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c17e76ca-983d-4914-bffb-aaec3a52ea51" containerName="korrel8r" Oct 30 11:18:11.269697 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.269604 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab9ecb55-83a5-4c09-a623-68419ffbf1c0" containerName="otc-container" Oct 30 11:18:11.269697 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.269616 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="c17e76ca-983d-4914-bffb-aaec3a52ea51" containerName="korrel8r" Oct 30 11:18:11.274075 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.274060 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-lx4r9" Oct 30 11:18:11.292318 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.292294 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-lx4r9"] Oct 30 11:18:11.302317 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.302292 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 11:18:11.302450 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.302405 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 11:18:11.302450 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.302412 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-8cj2g\"" Oct 30 11:18:11.302554 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.302473 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 11:18:11.302554 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.302485 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 11:18:11.354677 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.354649 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4l5t\" (UniqueName: \"kubernetes.io/projected/f5b90f06-f20c-4596-ada5-b22a9c5d45d2-kube-api-access-v4l5t\") pod \"korrel8r-6644769df5-lx4r9\" (UID: \"f5b90f06-f20c-4596-ada5-b22a9c5d45d2\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-lx4r9" Oct 30 11:18:11.354805 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.354697 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f5b90f06-f20c-4596-ada5-b22a9c5d45d2-tls\") pod \"korrel8r-6644769df5-lx4r9\" (UID: \"f5b90f06-f20c-4596-ada5-b22a9c5d45d2\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-lx4r9" Oct 30 11:18:11.354805 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.354765 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f5b90f06-f20c-4596-ada5-b22a9c5d45d2-config\") pod \"korrel8r-6644769df5-lx4r9\" (UID: \"f5b90f06-f20c-4596-ada5-b22a9c5d45d2\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-lx4r9" Oct 30 11:18:11.429347 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.429291 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:11.455212 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.455177 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4l5t\" (UniqueName: \"kubernetes.io/projected/f5b90f06-f20c-4596-ada5-b22a9c5d45d2-kube-api-access-v4l5t\") pod \"korrel8r-6644769df5-lx4r9\" (UID: \"f5b90f06-f20c-4596-ada5-b22a9c5d45d2\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-lx4r9" Oct 30 11:18:11.455352 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.455233 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f5b90f06-f20c-4596-ada5-b22a9c5d45d2-tls\") pod \"korrel8r-6644769df5-lx4r9\" (UID: \"f5b90f06-f20c-4596-ada5-b22a9c5d45d2\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-lx4r9" Oct 30 11:18:11.455352 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.455263 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f5b90f06-f20c-4596-ada5-b22a9c5d45d2-config\") pod \"korrel8r-6644769df5-lx4r9\" (UID: \"f5b90f06-f20c-4596-ada5-b22a9c5d45d2\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-lx4r9" Oct 30 11:18:11.456097 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.456065 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f5b90f06-f20c-4596-ada5-b22a9c5d45d2-config\") pod \"korrel8r-6644769df5-lx4r9\" (UID: \"f5b90f06-f20c-4596-ada5-b22a9c5d45d2\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-lx4r9" Oct 30 11:18:11.458017 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.457990 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f5b90f06-f20c-4596-ada5-b22a9c5d45d2-tls\") pod \"korrel8r-6644769df5-lx4r9\" (UID: \"f5b90f06-f20c-4596-ada5-b22a9c5d45d2\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-lx4r9" Oct 30 11:18:11.478597 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.478570 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4l5t\" (UniqueName: \"kubernetes.io/projected/f5b90f06-f20c-4596-ada5-b22a9c5d45d2-kube-api-access-v4l5t\") pod \"korrel8r-6644769df5-lx4r9\" (UID: \"f5b90f06-f20c-4596-ada5-b22a9c5d45d2\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-lx4r9" Oct 30 11:18:11.586768 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.586685 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-lx4r9" Oct 30 11:18:11.592007 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.591974 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:11.894580 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.893562 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-lx4r9"] Oct 30 11:18:11.894580 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:18:11.894265 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b90f06_f20c_4596_ada5_b22a9c5d45d2.slice/crio-e4dcd9b0d7b25c5176981b49222e1eb22e879058ab6e6412f6c982b4cb8d867c WatchSource:0}: Error finding container e4dcd9b0d7b25c5176981b49222e1eb22e879058ab6e6412f6c982b4cb8d867c: Status 404 returned error can't find the container with id e4dcd9b0d7b25c5176981b49222e1eb22e879058ab6e6412f6c982b4cb8d867c Oct 30 11:18:11.894858 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.894581 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 11:18:11.898957 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.898939 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:18:11.915882 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.915863 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-61f7caf9\"" Oct 30 11:18:11.918755 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.918607 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 11:18:11.924561 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.923942 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-prh6t\"" Oct 30 11:18:11.929791 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.929771 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 11:18:11.983870 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:11.983834 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:12.059043 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:12.059006 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw2vh\" (UniqueName: \"kubernetes.io/projected/5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617-kube-api-access-qw2vh\") pod \"data-science-collector-collector-0\" (UID: \"5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:18:12.059230 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:12.059088 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:18:12.059230 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:12.059115 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:18:12.160482 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:12.160382 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:18:12.160482 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:12.160422 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:18:12.160482 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:12.160482 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw2vh\" (UniqueName: \"kubernetes.io/projected/5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617-kube-api-access-qw2vh\") pod \"data-science-collector-collector-0\" (UID: \"5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:18:12.161110 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:12.161090 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:18:12.162765 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:12.162743 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:12.163114 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:12.163098 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:18:12.183691 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:12.183665 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw2vh\" (UniqueName: \"kubernetes.io/projected/5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617-kube-api-access-qw2vh\") pod \"data-science-collector-collector-0\" (UID: \"5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:18:12.217464 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:12.217443 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:18:12.232705 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:12.232678 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-lx4r9" event={"ID":"f5b90f06-f20c-4596-ada5-b22a9c5d45d2","Type":"ContainerStarted","Data":"534aaf2c4a4e0e769f208546af4c850dac83bbaa12faf322bf24c4d118aa6943"} Oct 30 11:18:12.232819 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:12.232726 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-lx4r9" event={"ID":"f5b90f06-f20c-4596-ada5-b22a9c5d45d2","Type":"ContainerStarted","Data":"e4dcd9b0d7b25c5176981b49222e1eb22e879058ab6e6412f6c982b4cb8d867c"} Oct 30 11:18:12.232871 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:12.232848 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-lx4r9" Oct 30 11:18:12.274267 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:12.274204 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-lx4r9" podStartSLOduration=1.274187665 podStartE2EDuration="1.274187665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:18:12.272827934 +0000 UTC m=+1982.774207420" watchObservedRunningTime="2026-10-30 11:18:12.274187665 +0000 UTC m=+1982.775567158" Oct 30 11:18:12.509997 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:12.509968 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 11:18:12.510467 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:18:12.510424 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d2c1d1a_1ba3_4085_be9b_3a7c2aabb617.slice/crio-f479dd81b1d619d2d4080f1a667aaf9877fec7d73427dd87f44dfc92777b50dc WatchSource:0}: Error finding container f479dd81b1d619d2d4080f1a667aaf9877fec7d73427dd87f44dfc92777b50dc: Status 404 returned error can't find the container with id f479dd81b1d619d2d4080f1a667aaf9877fec7d73427dd87f44dfc92777b50dc Oct 30 11:18:12.512713 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:12.512675 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:18:12.512804 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:12.512771 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:18:12.727873 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:12.727834 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:12.903846 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:12.903744 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:13.238324 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:13.238287 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617","Type":"ContainerStarted","Data":"a773d392aae01c631d9047c99407c35ffc15665cbecc437f7ee629227b20d1ad"} Oct 30 11:18:13.238508 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:13.238345 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617","Type":"ContainerStarted","Data":"f479dd81b1d619d2d4080f1a667aaf9877fec7d73427dd87f44dfc92777b50dc"} Oct 30 11:18:13.283892 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:13.283844 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.283831714 podStartE2EDuration="2.283831714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:18:13.282726948 +0000 UTC m=+1983.784106435" watchObservedRunningTime="2026-10-30 11:18:13.283831714 +0000 UTC m=+1983.785211194" Oct 30 11:18:23.241213 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:23.241176 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-lx4r9" Oct 30 11:18:26.256926 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:18:26.256891 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:18:29.109336 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.109301 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-lx4r9"] Oct 30 11:18:29.109791 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.109504 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-lx4r9" podUID="f5b90f06-f20c-4596-ada5-b22a9c5d45d2" containerName="korrel8r" containerID="cri-o://534aaf2c4a4e0e769f208546af4c850dac83bbaa12faf322bf24c4d118aa6943" gracePeriod=30 Oct 30 11:18:29.305705 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.305675 4289 generic.go:356] "Generic (PLEG): container finished" podID="f5b90f06-f20c-4596-ada5-b22a9c5d45d2" containerID="534aaf2c4a4e0e769f208546af4c850dac83bbaa12faf322bf24c4d118aa6943" exitCode=0 Oct 30 11:18:29.305851 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.305739 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-lx4r9" event={"ID":"f5b90f06-f20c-4596-ada5-b22a9c5d45d2","Type":"ContainerDied","Data":"534aaf2c4a4e0e769f208546af4c850dac83bbaa12faf322bf24c4d118aa6943"} Oct 30 11:18:29.357221 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.357197 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-lx4r9" Oct 30 11:18:29.390983 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.390926 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5b90f06-f20c-4596-ada5-b22a9c5d45d2-tls\" (UniqueName: \"kubernetes.io/secret/f5b90f06-f20c-4596-ada5-b22a9c5d45d2-tls\") pod \"f5b90f06-f20c-4596-ada5-b22a9c5d45d2\" (UID: \"f5b90f06-f20c-4596-ada5-b22a9c5d45d2\") " Oct 30 11:18:29.390983 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.390966 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b90f06-f20c-4596-ada5-b22a9c5d45d2-kube-api-access-v4l5t\" (UniqueName: \"kubernetes.io/projected/f5b90f06-f20c-4596-ada5-b22a9c5d45d2-kube-api-access-v4l5t\") pod \"f5b90f06-f20c-4596-ada5-b22a9c5d45d2\" (UID: \"f5b90f06-f20c-4596-ada5-b22a9c5d45d2\") " Oct 30 11:18:29.391173 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.390989 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b90f06-f20c-4596-ada5-b22a9c5d45d2-config\" (UniqueName: \"kubernetes.io/configmap/f5b90f06-f20c-4596-ada5-b22a9c5d45d2-config\") pod \"f5b90f06-f20c-4596-ada5-b22a9c5d45d2\" (UID: \"f5b90f06-f20c-4596-ada5-b22a9c5d45d2\") " Oct 30 11:18:29.391381 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.391360 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b90f06-f20c-4596-ada5-b22a9c5d45d2-config" pod "f5b90f06-f20c-4596-ada5-b22a9c5d45d2" (UID: "f5b90f06-f20c-4596-ada5-b22a9c5d45d2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:18:29.392980 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.392958 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b90f06-f20c-4596-ada5-b22a9c5d45d2-kube-api-access-v4l5t" pod "f5b90f06-f20c-4596-ada5-b22a9c5d45d2" (UID: "f5b90f06-f20c-4596-ada5-b22a9c5d45d2"). InnerVolumeSpecName "kube-api-access-v4l5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:18:29.393047 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.392989 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5b90f06-f20c-4596-ada5-b22a9c5d45d2-tls" pod "f5b90f06-f20c-4596-ada5-b22a9c5d45d2" (UID: "f5b90f06-f20c-4596-ada5-b22a9c5d45d2"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:18:29.492899 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.492865 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f5b90f06-f20c-4596-ada5-b22a9c5d45d2-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:18:29.492899 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.492899 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4l5t\" (UniqueName: \"kubernetes.io/projected/f5b90f06-f20c-4596-ada5-b22a9c5d45d2-kube-api-access-v4l5t\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:18:29.493149 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.492914 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f5b90f06-f20c-4596-ada5-b22a9c5d45d2-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:18:29.506299 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.506266 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 11:18:29.506604 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.506547 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617" containerName="otc-container" containerID="cri-o://a773d392aae01c631d9047c99407c35ffc15665cbecc437f7ee629227b20d1ad" gracePeriod=30 Oct 30 11:18:29.607922 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.607885 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:29.736973 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.736949 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:18:29.794878 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.794835 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617-monitoring-tls-certs\") pod \"5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617\" (UID: \"5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617\") " Oct 30 11:18:29.795057 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.794914 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617-kube-api-access-qw2vh\" (UniqueName: \"kubernetes.io/projected/5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617-kube-api-access-qw2vh\") pod \"5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617\" (UID: \"5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617\") " Oct 30 11:18:29.795057 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.794953 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617-otc-internal\") pod \"5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617\" (UID: \"5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617\") " Oct 30 11:18:29.795368 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.795345 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617-otc-internal" pod "5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617" (UID: "5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:18:29.797022 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.796991 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617-monitoring-tls-certs" pod "5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617" (UID: "5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:18:29.797657 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.797636 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617-kube-api-access-qw2vh" pod "5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617" (UID: "5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617"). InnerVolumeSpecName "kube-api-access-qw2vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:18:29.802522 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.802493 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:29.896446 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.896397 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:18:29.896446 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.896444 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw2vh\" (UniqueName: \"kubernetes.io/projected/5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617-kube-api-access-qw2vh\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:18:29.896632 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:29.896455 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:18:30.075523 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:30.075491 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:30.256838 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:30.256800 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:30.310626 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:30.310596 4289 generic.go:356] "Generic (PLEG): container finished" podID="5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617" containerID="a773d392aae01c631d9047c99407c35ffc15665cbecc437f7ee629227b20d1ad" exitCode=0 Oct 30 11:18:30.310822 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:30.310642 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:18:30.310822 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:30.310675 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617","Type":"ContainerDied","Data":"a773d392aae01c631d9047c99407c35ffc15665cbecc437f7ee629227b20d1ad"} Oct 30 11:18:30.310822 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:30.310725 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617","Type":"ContainerDied","Data":"f479dd81b1d619d2d4080f1a667aaf9877fec7d73427dd87f44dfc92777b50dc"} Oct 30 11:18:30.310822 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:30.310756 4289 scope.go:122] "RemoveContainer" containerID="a773d392aae01c631d9047c99407c35ffc15665cbecc437f7ee629227b20d1ad" Oct 30 11:18:30.312178 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:30.312154 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-lx4r9" event={"ID":"f5b90f06-f20c-4596-ada5-b22a9c5d45d2","Type":"ContainerDied","Data":"e4dcd9b0d7b25c5176981b49222e1eb22e879058ab6e6412f6c982b4cb8d867c"} Oct 30 11:18:30.312228 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:30.312190 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-lx4r9" Oct 30 11:18:30.319238 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:30.319219 4289 scope.go:122] "RemoveContainer" containerID="a773d392aae01c631d9047c99407c35ffc15665cbecc437f7ee629227b20d1ad" Oct 30 11:18:30.319505 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:18:30.319488 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a773d392aae01c631d9047c99407c35ffc15665cbecc437f7ee629227b20d1ad\": container with ID starting with a773d392aae01c631d9047c99407c35ffc15665cbecc437f7ee629227b20d1ad not found: ID does not exist" containerID="a773d392aae01c631d9047c99407c35ffc15665cbecc437f7ee629227b20d1ad" Oct 30 11:18:30.319551 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:30.319513 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a773d392aae01c631d9047c99407c35ffc15665cbecc437f7ee629227b20d1ad"} err="failed to get container status \"a773d392aae01c631d9047c99407c35ffc15665cbecc437f7ee629227b20d1ad\": rpc error: code = NotFound desc = could not find container \"a773d392aae01c631d9047c99407c35ffc15665cbecc437f7ee629227b20d1ad\": container with ID starting with a773d392aae01c631d9047c99407c35ffc15665cbecc437f7ee629227b20d1ad not found: ID does not exist" Oct 30 11:18:30.319551 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:30.319530 4289 scope.go:122] "RemoveContainer" containerID="534aaf2c4a4e0e769f208546af4c850dac83bbaa12faf322bf24c4d118aa6943" Oct 30 11:18:30.360421 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:30.360385 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-lx4r9"] Oct 30 11:18:30.372019 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:30.371966 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-lx4r9"] Oct 30 11:18:30.400312 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:30.400285 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 11:18:30.404691 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:30.404662 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 11:18:31.624342 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:31.624300 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617" path="/var/lib/kubelet/pods/5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617/volumes" Oct 30 11:18:31.624739 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:31.624692 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b90f06-f20c-4596-ada5-b22a9c5d45d2" path="/var/lib/kubelet/pods/f5b90f06-f20c-4596-ada5-b22a9c5d45d2/volumes" Oct 30 11:18:33.372371 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.372336 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:33.534462 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.534407 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:33.601479 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.601421 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q"] Oct 30 11:18:33.601772 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.601759 4289 cpu_manager.go:414] "containerMap: removing container" podUID="f5b90f06-f20c-4596-ada5-b22a9c5d45d2" containerName="korrel8r" Oct 30 11:18:33.601831 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.601773 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b90f06-f20c-4596-ada5-b22a9c5d45d2" containerName="korrel8r" Oct 30 11:18:33.601831 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.601782 4289 cpu_manager.go:414] "containerMap: removing container" podUID="5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617" containerName="otc-container" Oct 30 11:18:33.601831 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.601787 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617" containerName="otc-container" Oct 30 11:18:33.601831 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.601840 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b90f06-f20c-4596-ada5-b22a9c5d45d2" containerName="korrel8r" Oct 30 11:18:33.601831 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.601849 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d2c1d1a-1ba3-4085-be9b-3a7c2aabb617" containerName="otc-container" Oct 30 11:18:33.606176 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.606159 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" Oct 30 11:18:33.624504 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.624425 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 11:18:33.628814 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.628788 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-krm2q\" (UID: \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" Oct 30 11:18:33.628958 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.628824 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-krm2q\" (UID: \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" Oct 30 11:18:33.628958 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.628924 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-krm2q\" (UID: \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" Oct 30 11:18:33.629067 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.628994 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-krm2q\" (UID: \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" Oct 30 11:18:33.629118 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.629077 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9pvq\" (UniqueName: \"kubernetes.io/projected/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-kube-api-access-n9pvq\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-krm2q\" (UID: \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" Oct 30 11:18:33.633669 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.633648 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-bb7pp\"" Oct 30 11:18:33.634862 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.634840 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q"] Oct 30 11:18:33.636008 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.635989 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 11:18:33.636082 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.636056 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 11:18:33.636264 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.636248 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 11:18:33.636312 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.636296 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 11:18:33.636360 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.636296 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 11:18:33.647618 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.647597 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6"] Oct 30 11:18:33.651274 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.651255 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" Oct 30 11:18:33.669081 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.669060 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6"] Oct 30 11:18:33.693663 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.693638 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw"] Oct 30 11:18:33.697113 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.697096 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" Oct 30 11:18:33.709581 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.709561 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:33.720660 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.720637 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw"] Oct 30 11:18:33.726652 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.726617 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 11:18:33.726652 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.726629 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-sdwlz\"" Oct 30 11:18:33.726875 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.726858 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 11:18:33.726921 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.726902 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 11:18:33.726921 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.726911 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 11:18:33.726980 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.726937 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-dqv6n\"" Oct 30 11:18:33.727095 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.727081 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 11:18:33.729952 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.729926 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/8f4aed18-f28d-482c-a868-229cdadff6f3-client-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6\" (UID: \"8f4aed18-f28d-482c-a868-229cdadff6f3\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" Oct 30 11:18:33.730023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.729977 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8f4aed18-f28d-482c-a868-229cdadff6f3-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6\" (UID: \"8f4aed18-f28d-482c-a868-229cdadff6f3\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" Oct 30 11:18:33.730076 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.730050 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-krm2q\" (UID: \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" Oct 30 11:18:33.730119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.730081 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-krm2q\" (UID: \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" Oct 30 11:18:33.730119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.730107 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/8f4aed18-f28d-482c-a868-229cdadff6f3-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6\" (UID: \"8f4aed18-f28d-482c-a868-229cdadff6f3\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" Oct 30 11:18:33.730213 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.730126 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/51c34556-aa72-4eed-ac1c-eb3f4534a62a-config\") pod \"data-science-thanos-querier-proxy-7b56895dd5-bdwbw\" (UID: \"51c34556-aa72-4eed-ac1c-eb3f4534a62a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" Oct 30 11:18:33.730213 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:18:33.730148 4289 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 30 11:18:33.730313 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:18:33.730218 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-prometheus-client-cert podName:3ba78eac-b3ef-4dc0-adf4-37897ca5b735 nodeName:}" failed. No retries permitted until 2026-10-30 11:18:34.230200193 +0000 UTC m=+2004.731579671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-krm2q" (UID: "3ba78eac-b3ef-4dc0-adf4-37897ca5b735") : secret "prometheus-operated-tls" not found Oct 30 11:18:33.730313 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.730151 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-krm2q\" (UID: \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" Oct 30 11:18:33.730313 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.730283 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-krm2q\" (UID: \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" Oct 30 11:18:33.730493 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.730314 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f4aed18-f28d-482c-a868-229cdadff6f3-config\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6\" (UID: \"8f4aed18-f28d-482c-a868-229cdadff6f3\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" Oct 30 11:18:33.730493 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.730342 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgvcx\" (UniqueName: \"kubernetes.io/projected/8f4aed18-f28d-482c-a868-229cdadff6f3-kube-api-access-dgvcx\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6\" (UID: \"8f4aed18-f28d-482c-a868-229cdadff6f3\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" Oct 30 11:18:33.730493 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.730372 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9pvq\" (UniqueName: \"kubernetes.io/projected/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-kube-api-access-n9pvq\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-krm2q\" (UID: \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" Oct 30 11:18:33.730493 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.730398 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/51c34556-aa72-4eed-ac1c-eb3f4534a62a-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-bdwbw\" (UID: \"51c34556-aa72-4eed-ac1c-eb3f4534a62a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" Oct 30 11:18:33.730696 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.730616 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nwdj\" (UniqueName: \"kubernetes.io/projected/51c34556-aa72-4eed-ac1c-eb3f4534a62a-kube-api-access-6nwdj\") pod \"data-science-thanos-querier-proxy-7b56895dd5-bdwbw\" (UID: \"51c34556-aa72-4eed-ac1c-eb3f4534a62a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" Oct 30 11:18:33.731330 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.731308 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-krm2q\" (UID: \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" Oct 30 11:18:33.731703 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.731685 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74"] Oct 30 11:18:33.732732 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.732710 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-krm2q\" (UID: \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" Oct 30 11:18:33.733062 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.733040 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-krm2q\" (UID: \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" Oct 30 11:18:33.735468 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.735449 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74" Oct 30 11:18:33.755384 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.755362 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74"] Oct 30 11:18:33.758286 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.758264 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9pvq\" (UniqueName: \"kubernetes.io/projected/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-kube-api-access-n9pvq\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-krm2q\" (UID: \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" Oct 30 11:18:33.824494 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.824469 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 11:18:33.824640 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.824551 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-rql7x\"" Oct 30 11:18:33.827231 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.827218 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 11:18:33.831550 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.831534 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8f4aed18-f28d-482c-a868-229cdadff6f3-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6\" (UID: \"8f4aed18-f28d-482c-a868-229cdadff6f3\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" Oct 30 11:18:33.831604 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.831564 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbtlb\" (UniqueName: \"kubernetes.io/projected/e473baa6-a55e-4bce-acb3-267ef3c8ad7e-kube-api-access-gbtlb\") pod \"korrel8r-5cbcc89b8f-96d74\" (UID: \"e473baa6-a55e-4bce-acb3-267ef3c8ad7e\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74" Oct 30 11:18:33.831710 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.831692 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/8f4aed18-f28d-482c-a868-229cdadff6f3-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6\" (UID: \"8f4aed18-f28d-482c-a868-229cdadff6f3\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" Oct 30 11:18:33.831771 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.831737 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/51c34556-aa72-4eed-ac1c-eb3f4534a62a-config\") pod \"data-science-thanos-querier-proxy-7b56895dd5-bdwbw\" (UID: \"51c34556-aa72-4eed-ac1c-eb3f4534a62a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" Oct 30 11:18:33.831842 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.831821 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f4aed18-f28d-482c-a868-229cdadff6f3-config\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6\" (UID: \"8f4aed18-f28d-482c-a868-229cdadff6f3\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" Oct 30 11:18:33.831902 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.831857 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgvcx\" (UniqueName: \"kubernetes.io/projected/8f4aed18-f28d-482c-a868-229cdadff6f3-kube-api-access-dgvcx\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6\" (UID: \"8f4aed18-f28d-482c-a868-229cdadff6f3\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" Oct 30 11:18:33.831902 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.831891 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/51c34556-aa72-4eed-ac1c-eb3f4534a62a-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-bdwbw\" (UID: \"51c34556-aa72-4eed-ac1c-eb3f4534a62a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" Oct 30 11:18:33.831999 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.831921 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nwdj\" (UniqueName: \"kubernetes.io/projected/51c34556-aa72-4eed-ac1c-eb3f4534a62a-kube-api-access-6nwdj\") pod \"data-science-thanos-querier-proxy-7b56895dd5-bdwbw\" (UID: \"51c34556-aa72-4eed-ac1c-eb3f4534a62a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" Oct 30 11:18:33.832112 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.832084 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e473baa6-a55e-4bce-acb3-267ef3c8ad7e-tls\") pod \"korrel8r-5cbcc89b8f-96d74\" (UID: \"e473baa6-a55e-4bce-acb3-267ef3c8ad7e\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74" Oct 30 11:18:33.832189 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.832123 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e473baa6-a55e-4bce-acb3-267ef3c8ad7e-config\") pod \"korrel8r-5cbcc89b8f-96d74\" (UID: \"e473baa6-a55e-4bce-acb3-267ef3c8ad7e\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74" Oct 30 11:18:33.832189 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.832162 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/8f4aed18-f28d-482c-a868-229cdadff6f3-client-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6\" (UID: \"8f4aed18-f28d-482c-a868-229cdadff6f3\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" Oct 30 11:18:33.832529 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.832506 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/8f4aed18-f28d-482c-a868-229cdadff6f3-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6\" (UID: \"8f4aed18-f28d-482c-a868-229cdadff6f3\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" Oct 30 11:18:33.832615 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.832511 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/51c34556-aa72-4eed-ac1c-eb3f4534a62a-config\") pod \"data-science-thanos-querier-proxy-7b56895dd5-bdwbw\" (UID: \"51c34556-aa72-4eed-ac1c-eb3f4534a62a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" Oct 30 11:18:33.832615 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.832513 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f4aed18-f28d-482c-a868-229cdadff6f3-config\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6\" (UID: \"8f4aed18-f28d-482c-a868-229cdadff6f3\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" Oct 30 11:18:33.832722 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.832651 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/8f4aed18-f28d-482c-a868-229cdadff6f3-client-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6\" (UID: \"8f4aed18-f28d-482c-a868-229cdadff6f3\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" Oct 30 11:18:33.834021 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.834002 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8f4aed18-f28d-482c-a868-229cdadff6f3-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6\" (UID: \"8f4aed18-f28d-482c-a868-229cdadff6f3\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" Oct 30 11:18:33.834454 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.834411 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/51c34556-aa72-4eed-ac1c-eb3f4534a62a-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-bdwbw\" (UID: \"51c34556-aa72-4eed-ac1c-eb3f4534a62a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" Oct 30 11:18:33.855044 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.855021 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgvcx\" (UniqueName: \"kubernetes.io/projected/8f4aed18-f28d-482c-a868-229cdadff6f3-kube-api-access-dgvcx\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6\" (UID: \"8f4aed18-f28d-482c-a868-229cdadff6f3\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" Oct 30 11:18:33.855128 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.855047 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nwdj\" (UniqueName: \"kubernetes.io/projected/51c34556-aa72-4eed-ac1c-eb3f4534a62a-kube-api-access-6nwdj\") pod \"data-science-thanos-querier-proxy-7b56895dd5-bdwbw\" (UID: \"51c34556-aa72-4eed-ac1c-eb3f4534a62a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" Oct 30 11:18:33.877285 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.877230 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:33.933656 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.933631 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e473baa6-a55e-4bce-acb3-267ef3c8ad7e-tls\") pod \"korrel8r-5cbcc89b8f-96d74\" (UID: \"e473baa6-a55e-4bce-acb3-267ef3c8ad7e\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74" Oct 30 11:18:33.933771 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.933658 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e473baa6-a55e-4bce-acb3-267ef3c8ad7e-config\") pod \"korrel8r-5cbcc89b8f-96d74\" (UID: \"e473baa6-a55e-4bce-acb3-267ef3c8ad7e\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74" Oct 30 11:18:33.933771 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.933683 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbtlb\" (UniqueName: \"kubernetes.io/projected/e473baa6-a55e-4bce-acb3-267ef3c8ad7e-kube-api-access-gbtlb\") pod \"korrel8r-5cbcc89b8f-96d74\" (UID: \"e473baa6-a55e-4bce-acb3-267ef3c8ad7e\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74" Oct 30 11:18:33.933771 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:18:33.933764 4289 secret.go:189] Couldn't get secret redhat-ods-monitoring/korrel8r-tls: secret "korrel8r-tls" not found Oct 30 11:18:33.933949 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:18:33.933825 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e473baa6-a55e-4bce-acb3-267ef3c8ad7e-tls podName:e473baa6-a55e-4bce-acb3-267ef3c8ad7e nodeName:}" failed. No retries permitted until 2026-10-30 11:18:34.433809998 +0000 UTC m=+2004.935189466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/e473baa6-a55e-4bce-acb3-267ef3c8ad7e-tls") pod "korrel8r-5cbcc89b8f-96d74" (UID: "e473baa6-a55e-4bce-acb3-267ef3c8ad7e") : secret "korrel8r-tls" not found Oct 30 11:18:33.934416 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.934392 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e473baa6-a55e-4bce-acb3-267ef3c8ad7e-config\") pod \"korrel8r-5cbcc89b8f-96d74\" (UID: \"e473baa6-a55e-4bce-acb3-267ef3c8ad7e\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74" Oct 30 11:18:33.957093 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.957073 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbtlb\" (UniqueName: \"kubernetes.io/projected/e473baa6-a55e-4bce-acb3-267ef3c8ad7e-kube-api-access-gbtlb\") pod \"korrel8r-5cbcc89b8f-96d74\" (UID: \"e473baa6-a55e-4bce-acb3-267ef3c8ad7e\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74" Oct 30 11:18:33.968938 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:33.968914 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" Oct 30 11:18:34.013214 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:34.013177 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" Oct 30 11:18:34.237080 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:34.237049 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-krm2q\" (UID: \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" Oct 30 11:18:34.239376 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:34.239353 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-krm2q\" (UID: \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" Oct 30 11:18:34.261646 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:34.261618 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6"] Oct 30 11:18:34.262379 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:34.262355 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:34.262537 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:18:34.262502 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4aed18_f28d_482c_a868_229cdadff6f3.slice/crio-e34f5b694fad5ad3cef02a9ce0a7c6e8054646254dd9324113f4fcd5a588b1b0 WatchSource:0}: Error finding container e34f5b694fad5ad3cef02a9ce0a7c6e8054646254dd9324113f4fcd5a588b1b0: Status 404 returned error can't find the container with id e34f5b694fad5ad3cef02a9ce0a7c6e8054646254dd9324113f4fcd5a588b1b0 Oct 30 11:18:34.313914 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:34.313889 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw"] Oct 30 11:18:34.314583 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:18:34.314560 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c34556_aa72_4eed_ac1c_eb3f4534a62a.slice/crio-736d136facbfe0a82d8b870effea1528db0b17aad68769f638a48f4572095107 WatchSource:0}: Error finding container 736d136facbfe0a82d8b870effea1528db0b17aad68769f638a48f4572095107: Status 404 returned error can't find the container with id 736d136facbfe0a82d8b870effea1528db0b17aad68769f638a48f4572095107 Oct 30 11:18:34.331543 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:34.331514 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" event={"ID":"8f4aed18-f28d-482c-a868-229cdadff6f3","Type":"ContainerStarted","Data":"e264a1beb7f6c23f7211feebc067b749a80445b1233013b4b0939b484625c48d"} Oct 30 11:18:34.331666 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:34.331581 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" event={"ID":"8f4aed18-f28d-482c-a868-229cdadff6f3","Type":"ContainerStarted","Data":"e34f5b694fad5ad3cef02a9ce0a7c6e8054646254dd9324113f4fcd5a588b1b0"} Oct 30 11:18:34.332655 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:34.332627 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" event={"ID":"51c34556-aa72-4eed-ac1c-eb3f4534a62a","Type":"ContainerStarted","Data":"736d136facbfe0a82d8b870effea1528db0b17aad68769f638a48f4572095107"} Oct 30 11:18:34.439621 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:34.439596 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e473baa6-a55e-4bce-acb3-267ef3c8ad7e-tls\") pod \"korrel8r-5cbcc89b8f-96d74\" (UID: \"e473baa6-a55e-4bce-acb3-267ef3c8ad7e\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74" Oct 30 11:18:34.441890 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:34.441862 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e473baa6-a55e-4bce-acb3-267ef3c8ad7e-tls\") pod \"korrel8r-5cbcc89b8f-96d74\" (UID: \"e473baa6-a55e-4bce-acb3-267ef3c8ad7e\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74" Oct 30 11:18:34.458540 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:34.458475 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:34.518380 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:34.518354 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" Oct 30 11:18:34.656049 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:34.656016 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:34.656798 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:34.656774 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74" Oct 30 11:18:34.814878 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:34.814833 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q"] Oct 30 11:18:34.815249 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:18:34.815224 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba78eac_b3ef_4dc0_adf4_37897ca5b735.slice/crio-a69195966b7eabda9ab0f859e0171fdf468e452dddf99e634fbe948f8981afe6 WatchSource:0}: Error finding container a69195966b7eabda9ab0f859e0171fdf468e452dddf99e634fbe948f8981afe6: Status 404 returned error can't find the container with id a69195966b7eabda9ab0f859e0171fdf468e452dddf99e634fbe948f8981afe6 Oct 30 11:18:34.952663 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:34.952634 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74"] Oct 30 11:18:34.953877 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:18:34.953845 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode473baa6_a55e_4bce_acb3_267ef3c8ad7e.slice/crio-42d2addc01fd52657cecb7a1b1d9f3090a062c59b0996615257c86fca626dc5b WatchSource:0}: Error finding container 42d2addc01fd52657cecb7a1b1d9f3090a062c59b0996615257c86fca626dc5b: Status 404 returned error can't find the container with id 42d2addc01fd52657cecb7a1b1d9f3090a062c59b0996615257c86fca626dc5b Oct 30 11:18:35.231207 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:35.231169 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:35.337798 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:35.337762 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" event={"ID":"3ba78eac-b3ef-4dc0-adf4-37897ca5b735","Type":"ContainerStarted","Data":"a69195966b7eabda9ab0f859e0171fdf468e452dddf99e634fbe948f8981afe6"} Oct 30 11:18:35.339705 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:35.339674 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" event={"ID":"8f4aed18-f28d-482c-a868-229cdadff6f3","Type":"ContainerStarted","Data":"f99bec7464c63f8187c41178beb393ff71d21d25a8cc1e5e6e7dac0204c1bde0"} Oct 30 11:18:35.339846 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:35.339720 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" event={"ID":"8f4aed18-f28d-482c-a868-229cdadff6f3","Type":"ContainerStarted","Data":"550edb5e79119d463de7368eafed4d348a13f8ca5a0ac17abd7071a377faa3b6"} Oct 30 11:18:35.341757 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:35.341731 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" event={"ID":"51c34556-aa72-4eed-ac1c-eb3f4534a62a","Type":"ContainerStarted","Data":"ea043eb61217ce74a94bebcbafac8c45b0133216a9d0a6d69c271b80be084358"} Oct 30 11:18:35.341844 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:35.341800 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" event={"ID":"51c34556-aa72-4eed-ac1c-eb3f4534a62a","Type":"ContainerStarted","Data":"ca9b61f6b0fd9993abb1ed7c4a61294207e0e37731a02934fac96dac0ddf7a08"} Oct 30 11:18:35.341892 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:35.341842 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" event={"ID":"51c34556-aa72-4eed-ac1c-eb3f4534a62a","Type":"ContainerStarted","Data":"81319697799fa1982985be2d5b853401f685e90ba342616622202fa6aef473e7"} Oct 30 11:18:35.343194 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:35.343165 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74" event={"ID":"e473baa6-a55e-4bce-acb3-267ef3c8ad7e","Type":"ContainerStarted","Data":"e483c93f8b697ada1bfd9997c401ecaf63a7859191aaa2124db6ab3ad707583f"} Oct 30 11:18:35.343293 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:35.343212 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74" event={"ID":"e473baa6-a55e-4bce-acb3-267ef3c8ad7e","Type":"ContainerStarted","Data":"42d2addc01fd52657cecb7a1b1d9f3090a062c59b0996615257c86fca626dc5b"} Oct 30 11:18:35.343293 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:35.343259 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74" Oct 30 11:18:35.384214 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:35.384168 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" podStartSLOduration=2.384154583 podStartE2EDuration="2.384154583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:18:35.384048812 +0000 UTC m=+2005.885428307" watchObservedRunningTime="2026-10-30 11:18:35.384154583 +0000 UTC m=+2005.885534070" Oct 30 11:18:35.417665 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:35.417628 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:35.428250 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:35.428191 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" podStartSLOduration=2.428173126 podStartE2EDuration="2.428173126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:18:35.427577443 +0000 UTC m=+2005.928956928" watchObservedRunningTime="2026-10-30 11:18:35.428173126 +0000 UTC m=+2005.929552613" Oct 30 11:18:35.468831 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:35.468734 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74" podStartSLOduration=2.468720286 podStartE2EDuration="2.468720286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:18:35.466893837 +0000 UTC m=+2005.968273323" watchObservedRunningTime="2026-10-30 11:18:35.468720286 +0000 UTC m=+2005.970099785" Oct 30 11:18:36.350100 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:36.350062 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" event={"ID":"3ba78eac-b3ef-4dc0-adf4-37897ca5b735","Type":"ContainerStarted","Data":"23bf9a9d69b72e89c854df1e58b2566695814d2d7d2016dcfd609a084cfe3ead"} Oct 30 11:18:36.350397 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:36.350378 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" Oct 30 11:18:36.390289 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:36.390238 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" podStartSLOduration=3.390225359 podStartE2EDuration="3.390225359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:18:36.389924703 +0000 UTC m=+2006.891304189" watchObservedRunningTime="2026-10-30 11:18:36.390225359 +0000 UTC m=+2006.891604835" Oct 30 11:18:37.915088 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:37.915054 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:38.157652 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.157610 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:38.333106 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.333067 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:38.519296 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.519260 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 11:18:38.523364 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.523345 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:38.553340 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.553304 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 11:18:38.553340 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.553335 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 11:18:38.553606 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.553353 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 11:18:38.553606 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.553390 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 11:18:38.553606 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.553450 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-jrn7f\"" Oct 30 11:18:38.555479 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.555458 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 11:18:38.576028 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.576005 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:38.576130 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.576039 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:38.576130 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.576064 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:38.576130 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.576083 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:38.576275 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.576201 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p8cb\" (UniqueName: \"kubernetes.io/projected/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-kube-api-access-2p8cb\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:38.576275 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.576238 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:38.576386 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.576363 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:38.582051 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.582021 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:38.607417 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.607352 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:18:38.611687 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.611672 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.642704 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.642674 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 11:18:38.642907 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.642781 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:18:38.643063 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.643014 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 11:18:38.643063 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.642943 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 11:18:38.643063 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.642781 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 11:18:38.643197 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.643085 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 11:18:38.643197 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.643093 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-k7dqn\"" Oct 30 11:18:38.677207 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.677168 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:38.677207 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.677206 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ff8a282-f987-4c65-911d-b56e9ea5dd09-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.677411 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.677227 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3ff8a282-f987-4c65-911d-b56e9ea5dd09-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.677411 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.677255 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3ff8a282-f987-4c65-911d-b56e9ea5dd09-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.677411 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.677304 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85c49\" (UniqueName: \"kubernetes.io/projected/3ff8a282-f987-4c65-911d-b56e9ea5dd09-kube-api-access-85c49\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.677594 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.677492 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:38.677594 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.677534 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ff8a282-f987-4c65-911d-b56e9ea5dd09-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.677594 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.677575 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:38.677734 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.677634 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:38.677734 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.677665 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:38.677734 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.677697 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p8cb\" (UniqueName: \"kubernetes.io/projected/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-kube-api-access-2p8cb\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:38.677734 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.677726 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ff8a282-f987-4c65-911d-b56e9ea5dd09-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.677931 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.677751 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.677931 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.677788 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:38.677931 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.677810 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.677931 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.677835 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.677931 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.677860 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.677931 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.677897 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.677931 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.677920 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:38.678274 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.677933 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.678274 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.678069 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 11:18:38.678274 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.678105 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 11:18:38.678274 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.678186 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 11:18:38.680794 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.680733 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:38.680928 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.680908 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:38.681012 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.680958 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:38.681160 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.681134 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:38.684946 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.684924 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:38.701836 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.701814 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p8cb\" (UniqueName: \"kubernetes.io/projected/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-kube-api-access-2p8cb\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:38.748319 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.748286 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:18:38.778898 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.778866 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ff8a282-f987-4c65-911d-b56e9ea5dd09-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.778898 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.778898 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.779136 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.778922 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.779136 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.778943 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.779136 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.779057 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.779136 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.779117 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.779341 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.779146 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.779341 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.779212 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ff8a282-f987-4c65-911d-b56e9ea5dd09-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.779341 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.779250 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3ff8a282-f987-4c65-911d-b56e9ea5dd09-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.779341 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.779282 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3ff8a282-f987-4c65-911d-b56e9ea5dd09-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.779341 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.779307 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85c49\" (UniqueName: \"kubernetes.io/projected/3ff8a282-f987-4c65-911d-b56e9ea5dd09-kube-api-access-85c49\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.779614 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.779353 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ff8a282-f987-4c65-911d-b56e9ea5dd09-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.780111 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.780083 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ff8a282-f987-4c65-911d-b56e9ea5dd09-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.780294 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.780265 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3ff8a282-f987-4c65-911d-b56e9ea5dd09-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.780388 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.780369 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3ff8a282-f987-4c65-911d-b56e9ea5dd09-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.781953 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.781925 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.782422 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.782396 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.782707 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.782679 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.782825 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.782739 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ff8a282-f987-4c65-911d-b56e9ea5dd09-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.782938 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.782922 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ff8a282-f987-4c65-911d-b56e9ea5dd09-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.783251 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.783236 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.783301 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.783253 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.783983 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.783966 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.802880 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.802857 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85c49\" (UniqueName: \"kubernetes.io/projected/3ff8a282-f987-4c65-911d-b56e9ea5dd09-kube-api-access-85c49\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:38.841406 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:38.841379 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:39.161816 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:39.161776 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 11:18:39.162128 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:18:39.161774 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39bbe9d3_25a1_4d9b_8eca_dc33e1fd455d.slice/crio-8ba2a386fa63e9240f9117dca49803a7c55de6e8ca90012dafd9ce8032031dfc WatchSource:0}: Error finding container 8ba2a386fa63e9240f9117dca49803a7c55de6e8ca90012dafd9ce8032031dfc: Status 404 returned error can't find the container with id 8ba2a386fa63e9240f9117dca49803a7c55de6e8ca90012dafd9ce8032031dfc Oct 30 11:18:39.363979 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:39.363945 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d","Type":"ContainerStarted","Data":"8ba2a386fa63e9240f9117dca49803a7c55de6e8ca90012dafd9ce8032031dfc"} Oct 30 11:18:40.369766 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:40.369723 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d","Type":"ContainerStarted","Data":"e9dab2b1e95a7eaa39c06e4cb981e9435a37b92cd85caeebf1a43a377d5c0681"} Oct 30 11:18:43.360864 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:43.360834 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" Oct 30 11:18:43.384954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:43.384926 4289 generic.go:356] "Generic (PLEG): container finished" podID="39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d" containerID="e9dab2b1e95a7eaa39c06e4cb981e9435a37b92cd85caeebf1a43a377d5c0681" exitCode=0 Oct 30 11:18:43.385089 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:43.384986 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d","Type":"ContainerDied","Data":"e9dab2b1e95a7eaa39c06e4cb981e9435a37b92cd85caeebf1a43a377d5c0681"} Oct 30 11:18:44.393608 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:44.393575 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d","Type":"ContainerStarted","Data":"288302b0523fb8b3fc3761bf69b697aeda8bfb9c9b17c819cf5bc439fc2a6fce"} Oct 30 11:18:44.394045 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:44.393640 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d","Type":"ContainerStarted","Data":"446d7d8d01226fc914207861ccaf24083c827da98d224fbe74ca6bcb4689c130"} Oct 30 11:18:44.394045 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:44.393840 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:44.465282 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:44.465232 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=2.880137835 podStartE2EDuration="6.465218257s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.585080422s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:18:44.452528968 +0000 UTC m=+2014.953908453" watchObservedRunningTime="2026-10-30 11:18:44.465218257 +0000 UTC m=+2014.966597742" Oct 30 11:18:45.335495 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:45.335463 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:45.363583 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:45.363549 4289 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:45.436827 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:45.436794 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:45.437268 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:45.436939 4289 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:45.450945 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:45.450917 4289 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") DevicePath \"csi-ea06c66241a0a003a5daff5d9fc602f40e53768b00b104c5e4118b92d558c090\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:45.555735 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:45.555701 4289 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7342da0537aa528cc1545d2018ac3fd0382dfe4f670067e58cf8ba34bb891174/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:45.584032 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:45.584006 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:45.831127 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:45.831086 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:46.153903 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:18:46.153861 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff8a282_f987_4c65_911d_b56e9ea5dd09.slice/crio-b2b07c0bcb37e5b868f778136c4ec08eee231084e2147fa677930b4adac939ab WatchSource:0}: Error finding container b2b07c0bcb37e5b868f778136c4ec08eee231084e2147fa677930b4adac939ab: Status 404 returned error can't find the container with id b2b07c0bcb37e5b868f778136c4ec08eee231084e2147fa677930b4adac939ab Oct 30 11:18:46.154678 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:46.154659 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:18:46.353155 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:46.353124 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74" Oct 30 11:18:46.420702 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:46.420609 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"3ff8a282-f987-4c65-911d-b56e9ea5dd09","Type":"ContainerStarted","Data":"b2b07c0bcb37e5b868f778136c4ec08eee231084e2147fa677930b4adac939ab"} Oct 30 11:18:47.426859 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:47.426818 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"3ff8a282-f987-4c65-911d-b56e9ea5dd09","Type":"ContainerStarted","Data":"6c8c73f3012976ca88fe74582857a7159373e77c2e1f8ea481504d3b36a533cf"} Oct 30 11:18:50.400522 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:50.400498 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:18:50.441259 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:50.441235 4289 generic.go:356] "Generic (PLEG): container finished" podID="3ff8a282-f987-4c65-911d-b56e9ea5dd09" containerID="6c8c73f3012976ca88fe74582857a7159373e77c2e1f8ea481504d3b36a533cf" exitCode=0 Oct 30 11:18:50.441384 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:50.441302 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"3ff8a282-f987-4c65-911d-b56e9ea5dd09","Type":"ContainerDied","Data":"6c8c73f3012976ca88fe74582857a7159373e77c2e1f8ea481504d3b36a533cf"} Oct 30 11:18:51.448024 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:51.447939 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"3ff8a282-f987-4c65-911d-b56e9ea5dd09","Type":"ContainerStarted","Data":"deb60fcd295d51457107e09e69000da518ab9300141e985a5a0b26ebae061751"} Oct 30 11:18:51.448392 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:51.448030 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"3ff8a282-f987-4c65-911d-b56e9ea5dd09","Type":"ContainerStarted","Data":"3c74ee91d769889d58aba84262f48d31c80ce163392f6721505a6fa6d6a2801f"} Oct 30 11:18:51.448392 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:51.448057 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"3ff8a282-f987-4c65-911d-b56e9ea5dd09","Type":"ContainerStarted","Data":"1420776adc26b76c076fcb26b0435d0e8892bcfb9a12dc85939a3b6ff762a564"} Oct 30 11:18:51.522699 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:51.522646 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=9.837482529 podStartE2EDuration="13.522632482s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.685149953s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:18:51.510900636 +0000 UTC m=+2022.012280133" watchObservedRunningTime="2026-10-30 11:18:51.522632482 +0000 UTC m=+2022.024011967" Oct 30 11:18:55.831477 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:55.831374 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:18:56.259557 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:18:56.259519 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:18:57.310910 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:18:57.310854 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:00.831650 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:00.831618 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:19:00.836929 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:00.836909 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:19:01.497616 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:01.497585 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:19:06.751269 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:06.751236 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:06.948946 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:06.948901 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:07.121192 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:07.121098 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:08.673310 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:08.673279 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:08.844884 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:08.844849 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:25.676954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:25.676921 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:25.849383 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:25.849337 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:26.257935 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:19:26.257897 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:19:33.839529 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:33.839487 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:39.400652 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:39.400611 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:39.642313 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:39.642275 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:39.721322 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:39.721240 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx"] Oct 30 11:19:39.724769 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:39.724751 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx" Oct 30 11:19:39.744838 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:39.744812 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx"] Oct 30 11:19:39.803935 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:39.803897 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:39.812987 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:39.812961 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c2a9d0b3-17b4-47ca-aada-ef737bf02adc-config\") pod \"korrel8r-66d6f7f696-dp6nx\" (UID: \"c2a9d0b3-17b4-47ca-aada-ef737bf02adc\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx" Oct 30 11:19:39.813099 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:39.812992 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c2a9d0b3-17b4-47ca-aada-ef737bf02adc-tls\") pod \"korrel8r-66d6f7f696-dp6nx\" (UID: \"c2a9d0b3-17b4-47ca-aada-ef737bf02adc\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx" Oct 30 11:19:39.813099 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:39.813024 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlzqk\" (UniqueName: \"kubernetes.io/projected/c2a9d0b3-17b4-47ca-aada-ef737bf02adc-kube-api-access-xlzqk\") pod \"korrel8r-66d6f7f696-dp6nx\" (UID: \"c2a9d0b3-17b4-47ca-aada-ef737bf02adc\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx" Oct 30 11:19:39.913908 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:39.913873 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c2a9d0b3-17b4-47ca-aada-ef737bf02adc-config\") pod \"korrel8r-66d6f7f696-dp6nx\" (UID: \"c2a9d0b3-17b4-47ca-aada-ef737bf02adc\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx" Oct 30 11:19:39.913908 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:39.913908 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c2a9d0b3-17b4-47ca-aada-ef737bf02adc-tls\") pod \"korrel8r-66d6f7f696-dp6nx\" (UID: \"c2a9d0b3-17b4-47ca-aada-ef737bf02adc\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx" Oct 30 11:19:39.914119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:39.913965 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlzqk\" (UniqueName: \"kubernetes.io/projected/c2a9d0b3-17b4-47ca-aada-ef737bf02adc-kube-api-access-xlzqk\") pod \"korrel8r-66d6f7f696-dp6nx\" (UID: \"c2a9d0b3-17b4-47ca-aada-ef737bf02adc\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx" Oct 30 11:19:39.914719 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:39.914701 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c2a9d0b3-17b4-47ca-aada-ef737bf02adc-config\") pod \"korrel8r-66d6f7f696-dp6nx\" (UID: \"c2a9d0b3-17b4-47ca-aada-ef737bf02adc\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx" Oct 30 11:19:39.916361 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:39.916340 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c2a9d0b3-17b4-47ca-aada-ef737bf02adc-tls\") pod \"korrel8r-66d6f7f696-dp6nx\" (UID: \"c2a9d0b3-17b4-47ca-aada-ef737bf02adc\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx" Oct 30 11:19:39.936815 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:39.936785 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlzqk\" (UniqueName: \"kubernetes.io/projected/c2a9d0b3-17b4-47ca-aada-ef737bf02adc-kube-api-access-xlzqk\") pod \"korrel8r-66d6f7f696-dp6nx\" (UID: \"c2a9d0b3-17b4-47ca-aada-ef737bf02adc\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx" Oct 30 11:19:40.037202 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:40.037127 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx" Oct 30 11:19:40.247133 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:40.247091 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:40.332126 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:40.332097 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx"] Oct 30 11:19:40.333830 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:19:40.333804 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a9d0b3_17b4_47ca_aada_ef737bf02adc.slice/crio-9fd98a6e4d1e4a2de9f99e7d9b88e47da2cb2cd6496b00d1997bcf1bf5dfb5d4 WatchSource:0}: Error finding container 9fd98a6e4d1e4a2de9f99e7d9b88e47da2cb2cd6496b00d1997bcf1bf5dfb5d4: Status 404 returned error can't find the container with id 9fd98a6e4d1e4a2de9f99e7d9b88e47da2cb2cd6496b00d1997bcf1bf5dfb5d4 Oct 30 11:19:40.426994 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:40.426955 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:40.675357 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:40.675274 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx" event={"ID":"c2a9d0b3-17b4-47ca-aada-ef737bf02adc","Type":"ContainerStarted","Data":"3e3cff5fc6e1382aadeb3801967a5b539307c60ec615e5f3ed37faf5c038f736"} Oct 30 11:19:40.675357 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:40.675332 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx" event={"ID":"c2a9d0b3-17b4-47ca-aada-ef737bf02adc","Type":"ContainerStarted","Data":"9fd98a6e4d1e4a2de9f99e7d9b88e47da2cb2cd6496b00d1997bcf1bf5dfb5d4"} Oct 30 11:19:40.675600 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:40.675416 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx" Oct 30 11:19:40.715992 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:40.715949 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx" podStartSLOduration=1.715936287 podStartE2EDuration="1.715936287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:19:40.714712637 +0000 UTC m=+2071.216092123" watchObservedRunningTime="2026-10-30 11:19:40.715936287 +0000 UTC m=+2071.217315771" Oct 30 11:19:41.413447 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:41.413389 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:41.653781 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:41.653741 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:42.255699 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:42.255655 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:42.429173 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:42.429129 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:43.799129 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:43.799098 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:19:43.802882 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:43.802853 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:43.829537 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:43.829512 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-xn2fk\"" Oct 30 11:19:43.829537 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:43.829512 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 11:19:43.829693 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:43.829554 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 11:19:43.829781 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:43.829760 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 11:19:43.841467 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:43.841413 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-7c92b5b1\"" Oct 30 11:19:43.843336 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:43.843315 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:19:43.953227 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:43.953194 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:43.953227 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:43.953228 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:43.953418 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:43.953251 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvnxs\" (UniqueName: \"kubernetes.io/projected/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-kube-api-access-gvnxs\") pod \"data-science-collector-collector-1\" (UID: \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:43.953418 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:43.953388 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:43.953512 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:43.953416 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:44.053984 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:44.053907 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:44.053984 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:44.053939 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:44.053984 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:44.053965 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvnxs\" (UniqueName: \"kubernetes.io/projected/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-kube-api-access-gvnxs\") pod \"data-science-collector-collector-1\" (UID: \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:44.054209 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:44.054010 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:44.054209 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:44.054027 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:44.054775 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:44.054748 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:44.056448 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:44.056411 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:44.056587 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:44.056502 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:44.056688 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:44.056668 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:44.077078 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:44.077047 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvnxs\" (UniqueName: \"kubernetes.io/projected/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-kube-api-access-gvnxs\") pod \"data-science-collector-collector-1\" (UID: \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:44.115599 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:44.115569 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:44.409080 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:44.409053 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:19:44.409191 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:19:44.409170 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6dbf5b6_f457_40e7_8335_7edf316ef6ef.slice/crio-57ca8ece8631ea70174ce6abe50ef2d6d8afce064c27f1601a8a5a03f654953c WatchSource:0}: Error finding container 57ca8ece8631ea70174ce6abe50ef2d6d8afce064c27f1601a8a5a03f654953c: Status 404 returned error can't find the container with id 57ca8ece8631ea70174ce6abe50ef2d6d8afce064c27f1601a8a5a03f654953c Oct 30 11:19:44.411517 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:44.411485 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:19:44.411603 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:44.411545 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:19:44.694749 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:44.694667 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e6dbf5b6-f457-40e7-8335-7edf316ef6ef","Type":"ContainerStarted","Data":"e372b59ba78c97f77af2c7d090df18a26db3a52a002af989b7a6e396fd21f4fc"} Oct 30 11:19:44.694749 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:44.694720 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e6dbf5b6-f457-40e7-8335-7edf316ef6ef","Type":"ContainerStarted","Data":"57ca8ece8631ea70174ce6abe50ef2d6d8afce064c27f1601a8a5a03f654953c"} Oct 30 11:19:44.736381 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:44.736328 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.7363165029999998 podStartE2EDuration="1.736316503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:19:44.735477748 +0000 UTC m=+2075.236857234" watchObservedRunningTime="2026-10-30 11:19:44.736316503 +0000 UTC m=+2075.237695988" Oct 30 11:19:47.876889 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:47.876836 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:48.073826 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:48.073778 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:48.184228 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:48.184145 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx"] Oct 30 11:19:48.184379 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:48.184344 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx" podUID="c2a9d0b3-17b4-47ca-aada-ef737bf02adc" containerName="korrel8r" containerID="cri-o://3e3cff5fc6e1382aadeb3801967a5b539307c60ec615e5f3ed37faf5c038f736" gracePeriod=30 Oct 30 11:19:48.186249 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:48.186224 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx" Oct 30 11:19:48.247965 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:48.247935 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:48.427167 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:48.427141 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx" Oct 30 11:19:48.494733 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:48.494702 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2a9d0b3-17b4-47ca-aada-ef737bf02adc-config\" (UniqueName: \"kubernetes.io/configmap/c2a9d0b3-17b4-47ca-aada-ef737bf02adc-config\") pod \"c2a9d0b3-17b4-47ca-aada-ef737bf02adc\" (UID: \"c2a9d0b3-17b4-47ca-aada-ef737bf02adc\") " Oct 30 11:19:48.494913 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:48.494747 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2a9d0b3-17b4-47ca-aada-ef737bf02adc-tls\" (UniqueName: \"kubernetes.io/secret/c2a9d0b3-17b4-47ca-aada-ef737bf02adc-tls\") pod \"c2a9d0b3-17b4-47ca-aada-ef737bf02adc\" (UID: \"c2a9d0b3-17b4-47ca-aada-ef737bf02adc\") " Oct 30 11:19:48.494913 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:48.494785 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2a9d0b3-17b4-47ca-aada-ef737bf02adc-kube-api-access-xlzqk\" (UniqueName: \"kubernetes.io/projected/c2a9d0b3-17b4-47ca-aada-ef737bf02adc-kube-api-access-xlzqk\") pod \"c2a9d0b3-17b4-47ca-aada-ef737bf02adc\" (UID: \"c2a9d0b3-17b4-47ca-aada-ef737bf02adc\") " Oct 30 11:19:48.495163 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:48.495139 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a9d0b3-17b4-47ca-aada-ef737bf02adc-config" pod "c2a9d0b3-17b4-47ca-aada-ef737bf02adc" (UID: "c2a9d0b3-17b4-47ca-aada-ef737bf02adc"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:19:48.496875 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:48.496851 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2a9d0b3-17b4-47ca-aada-ef737bf02adc-tls" pod "c2a9d0b3-17b4-47ca-aada-ef737bf02adc" (UID: "c2a9d0b3-17b4-47ca-aada-ef737bf02adc"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:19:48.496969 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:48.496908 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a9d0b3-17b4-47ca-aada-ef737bf02adc-kube-api-access-xlzqk" pod "c2a9d0b3-17b4-47ca-aada-ef737bf02adc" (UID: "c2a9d0b3-17b4-47ca-aada-ef737bf02adc"). InnerVolumeSpecName "kube-api-access-xlzqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:19:48.596636 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:48.596502 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c2a9d0b3-17b4-47ca-aada-ef737bf02adc-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:19:48.596636 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:48.596532 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c2a9d0b3-17b4-47ca-aada-ef737bf02adc-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:19:48.596636 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:48.596544 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlzqk\" (UniqueName: \"kubernetes.io/projected/c2a9d0b3-17b4-47ca-aada-ef737bf02adc-kube-api-access-xlzqk\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:19:48.715847 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:48.715808 4289 generic.go:356] "Generic (PLEG): container finished" podID="c2a9d0b3-17b4-47ca-aada-ef737bf02adc" containerID="3e3cff5fc6e1382aadeb3801967a5b539307c60ec615e5f3ed37faf5c038f736" exitCode=0 Oct 30 11:19:48.716009 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:48.715862 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx" Oct 30 11:19:48.716009 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:48.715895 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx" event={"ID":"c2a9d0b3-17b4-47ca-aada-ef737bf02adc","Type":"ContainerDied","Data":"3e3cff5fc6e1382aadeb3801967a5b539307c60ec615e5f3ed37faf5c038f736"} Oct 30 11:19:48.716009 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:48.715945 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx" event={"ID":"c2a9d0b3-17b4-47ca-aada-ef737bf02adc","Type":"ContainerDied","Data":"9fd98a6e4d1e4a2de9f99e7d9b88e47da2cb2cd6496b00d1997bcf1bf5dfb5d4"} Oct 30 11:19:48.716009 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:48.715966 4289 scope.go:122] "RemoveContainer" containerID="3e3cff5fc6e1382aadeb3801967a5b539307c60ec615e5f3ed37faf5c038f736" Oct 30 11:19:48.724625 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:48.724605 4289 scope.go:122] "RemoveContainer" containerID="3e3cff5fc6e1382aadeb3801967a5b539307c60ec615e5f3ed37faf5c038f736" Oct 30 11:19:48.724880 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:19:48.724859 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e3cff5fc6e1382aadeb3801967a5b539307c60ec615e5f3ed37faf5c038f736\": container with ID starting with 3e3cff5fc6e1382aadeb3801967a5b539307c60ec615e5f3ed37faf5c038f736 not found: ID does not exist" containerID="3e3cff5fc6e1382aadeb3801967a5b539307c60ec615e5f3ed37faf5c038f736" Oct 30 11:19:48.724946 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:48.724891 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e3cff5fc6e1382aadeb3801967a5b539307c60ec615e5f3ed37faf5c038f736"} err="failed to get container status \"3e3cff5fc6e1382aadeb3801967a5b539307c60ec615e5f3ed37faf5c038f736\": rpc error: code = NotFound desc = could not find container \"3e3cff5fc6e1382aadeb3801967a5b539307c60ec615e5f3ed37faf5c038f736\": container with ID starting with 3e3cff5fc6e1382aadeb3801967a5b539307c60ec615e5f3ed37faf5c038f736 not found: ID does not exist" Oct 30 11:19:48.764712 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:48.764650 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx"] Oct 30 11:19:48.776359 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:48.776337 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-dp6nx"] Oct 30 11:19:49.497772 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:49.497741 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:19:49.498174 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:49.497908 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="e6dbf5b6-f457-40e7-8335-7edf316ef6ef" containerName="otc-container" containerID="cri-o://e372b59ba78c97f77af2c7d090df18a26db3a52a002af989b7a6e396fd21f4fc" gracePeriod=30 Oct 30 11:19:49.625154 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:49.625127 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2a9d0b3-17b4-47ca-aada-ef737bf02adc" path="/var/lib/kubelet/pods/c2a9d0b3-17b4-47ca-aada-ef737bf02adc/volumes" Oct 30 11:19:49.687947 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:49.687911 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:49.723922 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:49.723890 4289 generic.go:356] "Generic (PLEG): container finished" podID="e6dbf5b6-f457-40e7-8335-7edf316ef6ef" containerID="e372b59ba78c97f77af2c7d090df18a26db3a52a002af989b7a6e396fd21f4fc" exitCode=0 Oct 30 11:19:49.724048 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:49.723957 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e6dbf5b6-f457-40e7-8335-7edf316ef6ef","Type":"ContainerDied","Data":"e372b59ba78c97f77af2c7d090df18a26db3a52a002af989b7a6e396fd21f4fc"} Oct 30 11:19:49.749282 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:49.749230 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:49.808325 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:49.808295 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-data-science-collector-ta-client-cert\") pod \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\" (UID: \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\") " Oct 30 11:19:49.808514 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:49.808355 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-monitoring-tls-certs\") pod \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\" (UID: \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\") " Oct 30 11:19:49.808514 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:49.808386 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-tls-certs\") pod \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\" (UID: \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\") " Oct 30 11:19:49.808514 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:49.808506 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-kube-api-access-gvnxs\" (UniqueName: \"kubernetes.io/projected/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-kube-api-access-gvnxs\") pod \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\" (UID: \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\") " Oct 30 11:19:49.808698 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:49.808540 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-otc-internal\" (UniqueName: \"kubernetes.io/configmap/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-otc-internal\") pod \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\" (UID: \"e6dbf5b6-f457-40e7-8335-7edf316ef6ef\") " Oct 30 11:19:49.809144 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:49.809115 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-otc-internal" pod "e6dbf5b6-f457-40e7-8335-7edf316ef6ef" (UID: "e6dbf5b6-f457-40e7-8335-7edf316ef6ef"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:19:49.810676 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:49.810648 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-monitoring-tls-certs" pod "e6dbf5b6-f457-40e7-8335-7edf316ef6ef" (UID: "e6dbf5b6-f457-40e7-8335-7edf316ef6ef"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:19:49.810890 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:49.810869 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-kube-api-access-gvnxs" pod "e6dbf5b6-f457-40e7-8335-7edf316ef6ef" (UID: "e6dbf5b6-f457-40e7-8335-7edf316ef6ef"). InnerVolumeSpecName "kube-api-access-gvnxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:19:49.811017 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:49.810993 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-data-science-collector-ta-client-cert" pod "e6dbf5b6-f457-40e7-8335-7edf316ef6ef" (UID: "e6dbf5b6-f457-40e7-8335-7edf316ef6ef"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:19:49.811153 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:49.811125 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-tls-certs" pod "e6dbf5b6-f457-40e7-8335-7edf316ef6ef" (UID: "e6dbf5b6-f457-40e7-8335-7edf316ef6ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:19:49.855805 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:49.855776 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:49.909649 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:49.909620 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:19:49.909649 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:49.909647 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:19:49.909833 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:49.909659 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:19:49.909833 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:49.909668 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvnxs\" (UniqueName: \"kubernetes.io/projected/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-kube-api-access-gvnxs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:19:49.909833 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:49.909676 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e6dbf5b6-f457-40e7-8335-7edf316ef6ef-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:19:50.556653 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.556616 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:50.729803 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.729768 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e6dbf5b6-f457-40e7-8335-7edf316ef6ef","Type":"ContainerDied","Data":"57ca8ece8631ea70174ce6abe50ef2d6d8afce064c27f1601a8a5a03f654953c"} Oct 30 11:19:50.729997 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.729822 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:50.729997 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.729835 4289 scope.go:122] "RemoveContainer" containerID="e372b59ba78c97f77af2c7d090df18a26db3a52a002af989b7a6e396fd21f4fc" Oct 30 11:19:50.779701 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.779674 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:19:50.791602 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.791574 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:19:50.822401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.822333 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:19:50.822740 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.822726 4289 cpu_manager.go:414] "containerMap: removing container" podUID="c2a9d0b3-17b4-47ca-aada-ef737bf02adc" containerName="korrel8r" Oct 30 11:19:50.822793 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.822742 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2a9d0b3-17b4-47ca-aada-ef737bf02adc" containerName="korrel8r" Oct 30 11:19:50.822793 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.822755 4289 cpu_manager.go:414] "containerMap: removing container" podUID="e6dbf5b6-f457-40e7-8335-7edf316ef6ef" containerName="otc-container" Oct 30 11:19:50.822793 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.822760 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6dbf5b6-f457-40e7-8335-7edf316ef6ef" containerName="otc-container" Oct 30 11:19:50.822923 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.822836 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2a9d0b3-17b4-47ca-aada-ef737bf02adc" containerName="korrel8r" Oct 30 11:19:50.822923 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.822845 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6dbf5b6-f457-40e7-8335-7edf316ef6ef" containerName="otc-container" Oct 30 11:19:50.827091 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.827072 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:50.844812 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.844772 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 11:19:50.846948 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.846918 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:19:50.847677 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.847658 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-xn2fk\"" Oct 30 11:19:50.847898 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.847871 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-e4cadd38\"" Oct 30 11:19:50.848011 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.847928 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 11:19:50.856094 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.856074 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 11:19:50.920748 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.920714 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2dc7690a-2e29-4ca4-a520-8de5afa045f9-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:50.920924 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.920755 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2dc7690a-2e29-4ca4-a520-8de5afa045f9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:50.920924 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.920775 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc7690a-2e29-4ca4-a520-8de5afa045f9-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:50.920924 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.920816 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd7fb\" (UniqueName: \"kubernetes.io/projected/2dc7690a-2e29-4ca4-a520-8de5afa045f9-kube-api-access-wd7fb\") pod \"data-science-collector-collector-1\" (UID: \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:50.920924 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:50.920912 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc7690a-2e29-4ca4-a520-8de5afa045f9-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:51.022171 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:51.022136 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2dc7690a-2e29-4ca4-a520-8de5afa045f9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:51.022171 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:51.022171 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc7690a-2e29-4ca4-a520-8de5afa045f9-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:51.022419 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:51.022191 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd7fb\" (UniqueName: \"kubernetes.io/projected/2dc7690a-2e29-4ca4-a520-8de5afa045f9-kube-api-access-wd7fb\") pod \"data-science-collector-collector-1\" (UID: \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:51.022419 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:51.022234 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc7690a-2e29-4ca4-a520-8de5afa045f9-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:51.022419 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:51.022358 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2dc7690a-2e29-4ca4-a520-8de5afa045f9-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:51.023007 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:51.022984 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2dc7690a-2e29-4ca4-a520-8de5afa045f9-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:51.024661 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:51.024640 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc7690a-2e29-4ca4-a520-8de5afa045f9-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:51.024745 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:51.024683 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc7690a-2e29-4ca4-a520-8de5afa045f9-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:51.024870 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:51.024850 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2dc7690a-2e29-4ca4-a520-8de5afa045f9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:51.044877 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:51.044854 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd7fb\" (UniqueName: \"kubernetes.io/projected/2dc7690a-2e29-4ca4-a520-8de5afa045f9-kube-api-access-wd7fb\") pod \"data-science-collector-collector-1\" (UID: \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:51.141767 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:51.141686 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:51.436182 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:51.436154 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:19:51.436409 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:19:51.436388 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc7690a_2e29_4ca4_a520_8de5afa045f9.slice/crio-c50526dca60e38adc02084732646d11b2fc0d717a9d941a84348237bc46c396d WatchSource:0}: Error finding container c50526dca60e38adc02084732646d11b2fc0d717a9d941a84348237bc46c396d: Status 404 returned error can't find the container with id c50526dca60e38adc02084732646d11b2fc0d717a9d941a84348237bc46c396d Oct 30 11:19:51.438468 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:51.438419 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:19:51.438548 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:51.438496 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:19:51.624405 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:51.624369 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6dbf5b6-f457-40e7-8335-7edf316ef6ef" path="/var/lib/kubelet/pods/e6dbf5b6-f457-40e7-8335-7edf316ef6ef/volumes" Oct 30 11:19:51.737317 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:51.737280 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2dc7690a-2e29-4ca4-a520-8de5afa045f9","Type":"ContainerStarted","Data":"340a4398f126f05d0edf5f736a27b0b3016af8ad1af57df7d6227967d8f955fd"} Oct 30 11:19:51.737509 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:51.737368 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2dc7690a-2e29-4ca4-a520-8de5afa045f9","Type":"ContainerStarted","Data":"c50526dca60e38adc02084732646d11b2fc0d717a9d941a84348237bc46c396d"} Oct 30 11:19:51.779316 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:51.779257 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.779243618 podStartE2EDuration="1.779243618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:19:51.778491838 +0000 UTC m=+2082.279871325" watchObservedRunningTime="2026-10-30 11:19:51.779243618 +0000 UTC m=+2082.280623102" Oct 30 11:19:56.257096 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:19:56.257055 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:19:56.627449 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:56.627322 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:56.844530 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:56.844490 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:56.980935 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:56.980898 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6"] Oct 30 11:19:56.984567 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:56.984543 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6" Oct 30 11:19:57.003707 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:57.003681 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6"] Oct 30 11:19:57.017785 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:57.017755 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:57.073975 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:57.073945 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2dvc\" (UniqueName: \"kubernetes.io/projected/b1c4f084-bf3c-4108-9784-874f5ec8fc22-kube-api-access-x2dvc\") pod \"korrel8r-66d6f7f696-stxz6\" (UID: \"b1c4f084-bf3c-4108-9784-874f5ec8fc22\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6" Oct 30 11:19:57.074119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:57.073981 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b1c4f084-bf3c-4108-9784-874f5ec8fc22-tls\") pod \"korrel8r-66d6f7f696-stxz6\" (UID: \"b1c4f084-bf3c-4108-9784-874f5ec8fc22\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6" Oct 30 11:19:57.074119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:57.074102 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1c4f084-bf3c-4108-9784-874f5ec8fc22-config\") pod \"korrel8r-66d6f7f696-stxz6\" (UID: \"b1c4f084-bf3c-4108-9784-874f5ec8fc22\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6" Oct 30 11:19:57.174759 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:57.174727 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1c4f084-bf3c-4108-9784-874f5ec8fc22-config\") pod \"korrel8r-66d6f7f696-stxz6\" (UID: \"b1c4f084-bf3c-4108-9784-874f5ec8fc22\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6" Oct 30 11:19:57.174911 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:57.174793 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2dvc\" (UniqueName: \"kubernetes.io/projected/b1c4f084-bf3c-4108-9784-874f5ec8fc22-kube-api-access-x2dvc\") pod \"korrel8r-66d6f7f696-stxz6\" (UID: \"b1c4f084-bf3c-4108-9784-874f5ec8fc22\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6" Oct 30 11:19:57.174911 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:57.174815 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b1c4f084-bf3c-4108-9784-874f5ec8fc22-tls\") pod \"korrel8r-66d6f7f696-stxz6\" (UID: \"b1c4f084-bf3c-4108-9784-874f5ec8fc22\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6" Oct 30 11:19:57.175506 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:57.175485 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1c4f084-bf3c-4108-9784-874f5ec8fc22-config\") pod \"korrel8r-66d6f7f696-stxz6\" (UID: \"b1c4f084-bf3c-4108-9784-874f5ec8fc22\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6" Oct 30 11:19:57.177374 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:57.177346 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b1c4f084-bf3c-4108-9784-874f5ec8fc22-tls\") pod \"korrel8r-66d6f7f696-stxz6\" (UID: \"b1c4f084-bf3c-4108-9784-874f5ec8fc22\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6" Oct 30 11:19:57.197262 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:57.197241 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2dvc\" (UniqueName: \"kubernetes.io/projected/b1c4f084-bf3c-4108-9784-874f5ec8fc22-kube-api-access-x2dvc\") pod \"korrel8r-66d6f7f696-stxz6\" (UID: \"b1c4f084-bf3c-4108-9784-874f5ec8fc22\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6" Oct 30 11:19:57.298016 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:57.297930 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6" Oct 30 11:19:57.428867 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:57.428822 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:57.591367 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:57.591322 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6"] Oct 30 11:19:57.591579 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:19:57.591557 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c4f084_bf3c_4108_9784_874f5ec8fc22.slice/crio-7c784cc5fa097ec739d7116b10ff3fee0ffe43fa218edf192f80161b39b2b7dc WatchSource:0}: Error finding container 7c784cc5fa097ec739d7116b10ff3fee0ffe43fa218edf192f80161b39b2b7dc: Status 404 returned error can't find the container with id 7c784cc5fa097ec739d7116b10ff3fee0ffe43fa218edf192f80161b39b2b7dc Oct 30 11:19:57.772735 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:57.772698 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6" event={"ID":"b1c4f084-bf3c-4108-9784-874f5ec8fc22","Type":"ContainerStarted","Data":"687c8025f6dc0f1c631be0aaca4d96722200b15bd36e6559c5afc6d411c41dc0"} Oct 30 11:19:57.772901 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:57.772756 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6" event={"ID":"b1c4f084-bf3c-4108-9784-874f5ec8fc22","Type":"ContainerStarted","Data":"7c784cc5fa097ec739d7116b10ff3fee0ffe43fa218edf192f80161b39b2b7dc"} Oct 30 11:19:57.772901 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:57.772785 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6" Oct 30 11:19:57.813832 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:57.813781 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6" podStartSLOduration=1.813763107 podStartE2EDuration="1.813763107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:19:57.812117314 +0000 UTC m=+2088.313496802" watchObservedRunningTime="2026-10-30 11:19:57.813763107 +0000 UTC m=+2088.315142591" Oct 30 11:19:58.359119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.359088 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:19:58.359536 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.359260 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="2dc7690a-2e29-4ca4-a520-8de5afa045f9" containerName="otc-container" containerID="cri-o://340a4398f126f05d0edf5f736a27b0b3016af8ad1af57df7d6227967d8f955fd" gracePeriod=30 Oct 30 11:19:58.538832 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.538782 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:58.605121 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.605098 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:58.687744 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.687682 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dc7690a-2e29-4ca4-a520-8de5afa045f9-otc-internal\" (UniqueName: \"kubernetes.io/configmap/2dc7690a-2e29-4ca4-a520-8de5afa045f9-otc-internal\") pod \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\" (UID: \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\") " Oct 30 11:19:58.687892 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.687744 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dc7690a-2e29-4ca4-a520-8de5afa045f9-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2dc7690a-2e29-4ca4-a520-8de5afa045f9-data-science-collector-ta-client-cert\") pod \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\" (UID: \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\") " Oct 30 11:19:58.687892 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.687852 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dc7690a-2e29-4ca4-a520-8de5afa045f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc7690a-2e29-4ca4-a520-8de5afa045f9-tls-certs\") pod \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\" (UID: \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\") " Oct 30 11:19:58.687979 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.687908 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dc7690a-2e29-4ca4-a520-8de5afa045f9-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc7690a-2e29-4ca4-a520-8de5afa045f9-monitoring-tls-certs\") pod \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\" (UID: \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\") " Oct 30 11:19:58.688033 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.687976 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc7690a-2e29-4ca4-a520-8de5afa045f9-kube-api-access-wd7fb\" (UniqueName: \"kubernetes.io/projected/2dc7690a-2e29-4ca4-a520-8de5afa045f9-kube-api-access-wd7fb\") pod \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\" (UID: \"2dc7690a-2e29-4ca4-a520-8de5afa045f9\") " Oct 30 11:19:58.688095 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.688075 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dc7690a-2e29-4ca4-a520-8de5afa045f9-otc-internal" pod "2dc7690a-2e29-4ca4-a520-8de5afa045f9" (UID: "2dc7690a-2e29-4ca4-a520-8de5afa045f9"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:19:58.688314 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.688295 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2dc7690a-2e29-4ca4-a520-8de5afa045f9-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:19:58.689974 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.689944 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dc7690a-2e29-4ca4-a520-8de5afa045f9-data-science-collector-ta-client-cert" pod "2dc7690a-2e29-4ca4-a520-8de5afa045f9" (UID: "2dc7690a-2e29-4ca4-a520-8de5afa045f9"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:19:58.690087 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.690018 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dc7690a-2e29-4ca4-a520-8de5afa045f9-tls-certs" pod "2dc7690a-2e29-4ca4-a520-8de5afa045f9" (UID: "2dc7690a-2e29-4ca4-a520-8de5afa045f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:19:58.690147 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.690099 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc7690a-2e29-4ca4-a520-8de5afa045f9-kube-api-access-wd7fb" pod "2dc7690a-2e29-4ca4-a520-8de5afa045f9" (UID: "2dc7690a-2e29-4ca4-a520-8de5afa045f9"). InnerVolumeSpecName "kube-api-access-wd7fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:19:58.690244 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.690228 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dc7690a-2e29-4ca4-a520-8de5afa045f9-monitoring-tls-certs" pod "2dc7690a-2e29-4ca4-a520-8de5afa045f9" (UID: "2dc7690a-2e29-4ca4-a520-8de5afa045f9"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:19:58.748990 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.748952 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:58.778318 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.778287 4289 generic.go:356] "Generic (PLEG): container finished" podID="2dc7690a-2e29-4ca4-a520-8de5afa045f9" containerID="340a4398f126f05d0edf5f736a27b0b3016af8ad1af57df7d6227967d8f955fd" exitCode=0 Oct 30 11:19:58.778516 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.778334 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:19:58.778516 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.778364 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2dc7690a-2e29-4ca4-a520-8de5afa045f9","Type":"ContainerDied","Data":"340a4398f126f05d0edf5f736a27b0b3016af8ad1af57df7d6227967d8f955fd"} Oct 30 11:19:58.778516 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.778424 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2dc7690a-2e29-4ca4-a520-8de5afa045f9","Type":"ContainerDied","Data":"c50526dca60e38adc02084732646d11b2fc0d717a9d941a84348237bc46c396d"} Oct 30 11:19:58.778516 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.778459 4289 scope.go:122] "RemoveContainer" containerID="340a4398f126f05d0edf5f736a27b0b3016af8ad1af57df7d6227967d8f955fd" Oct 30 11:19:58.787424 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.787406 4289 scope.go:122] "RemoveContainer" containerID="340a4398f126f05d0edf5f736a27b0b3016af8ad1af57df7d6227967d8f955fd" Oct 30 11:19:58.787668 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:19:58.787646 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"340a4398f126f05d0edf5f736a27b0b3016af8ad1af57df7d6227967d8f955fd\": container with ID starting with 340a4398f126f05d0edf5f736a27b0b3016af8ad1af57df7d6227967d8f955fd not found: ID does not exist" containerID="340a4398f126f05d0edf5f736a27b0b3016af8ad1af57df7d6227967d8f955fd" Oct 30 11:19:58.787733 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.787675 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340a4398f126f05d0edf5f736a27b0b3016af8ad1af57df7d6227967d8f955fd"} err="failed to get container status \"340a4398f126f05d0edf5f736a27b0b3016af8ad1af57df7d6227967d8f955fd\": rpc error: code = NotFound desc = could not find container \"340a4398f126f05d0edf5f736a27b0b3016af8ad1af57df7d6227967d8f955fd\": container with ID starting with 340a4398f126f05d0edf5f736a27b0b3016af8ad1af57df7d6227967d8f955fd not found: ID does not exist" Oct 30 11:19:58.789100 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.789078 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2dc7690a-2e29-4ca4-a520-8de5afa045f9-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:19:58.789229 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.789103 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc7690a-2e29-4ca4-a520-8de5afa045f9-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:19:58.789229 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.789118 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc7690a-2e29-4ca4-a520-8de5afa045f9-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:19:58.789229 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.789131 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd7fb\" (UniqueName: \"kubernetes.io/projected/2dc7690a-2e29-4ca4-a520-8de5afa045f9-kube-api-access-wd7fb\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:19:58.828116 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.828092 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:19:58.832475 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:58.832453 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:19:59.566513 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:59.566483 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:19:59.624601 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:59.624571 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dc7690a-2e29-4ca4-a520-8de5afa045f9" path="/var/lib/kubelet/pods/2dc7690a-2e29-4ca4-a520-8de5afa045f9/volumes" Oct 30 11:19:59.724176 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:19:59.724138 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:04.939300 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:04.939263 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:05.137539 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:05.137499 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:05.253906 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:05.253872 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6"] Oct 30 11:20:05.254102 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:05.254077 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6" podUID="b1c4f084-bf3c-4108-9784-874f5ec8fc22" containerName="korrel8r" containerID="cri-o://687c8025f6dc0f1c631be0aaca4d96722200b15bd36e6559c5afc6d411c41dc0" gracePeriod=30 Oct 30 11:20:05.256053 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:05.256030 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6" Oct 30 11:20:05.310176 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:05.310145 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:05.509367 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:05.509308 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6" Oct 30 11:20:05.550005 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:05.549975 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1c4f084-bf3c-4108-9784-874f5ec8fc22-config\" (UniqueName: \"kubernetes.io/configmap/b1c4f084-bf3c-4108-9784-874f5ec8fc22-config\") pod \"b1c4f084-bf3c-4108-9784-874f5ec8fc22\" (UID: \"b1c4f084-bf3c-4108-9784-874f5ec8fc22\") " Oct 30 11:20:05.550158 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:05.550036 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1c4f084-bf3c-4108-9784-874f5ec8fc22-kube-api-access-x2dvc\" (UniqueName: \"kubernetes.io/projected/b1c4f084-bf3c-4108-9784-874f5ec8fc22-kube-api-access-x2dvc\") pod \"b1c4f084-bf3c-4108-9784-874f5ec8fc22\" (UID: \"b1c4f084-bf3c-4108-9784-874f5ec8fc22\") " Oct 30 11:20:05.550158 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:05.550076 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1c4f084-bf3c-4108-9784-874f5ec8fc22-tls\" (UniqueName: \"kubernetes.io/secret/b1c4f084-bf3c-4108-9784-874f5ec8fc22-tls\") pod \"b1c4f084-bf3c-4108-9784-874f5ec8fc22\" (UID: \"b1c4f084-bf3c-4108-9784-874f5ec8fc22\") " Oct 30 11:20:05.550576 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:05.550544 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1c4f084-bf3c-4108-9784-874f5ec8fc22-config" pod "b1c4f084-bf3c-4108-9784-874f5ec8fc22" (UID: "b1c4f084-bf3c-4108-9784-874f5ec8fc22"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:20:05.552566 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:05.552538 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1c4f084-bf3c-4108-9784-874f5ec8fc22-tls" pod "b1c4f084-bf3c-4108-9784-874f5ec8fc22" (UID: "b1c4f084-bf3c-4108-9784-874f5ec8fc22"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:20:05.553547 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:05.552861 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c4f084-bf3c-4108-9784-874f5ec8fc22-kube-api-access-x2dvc" pod "b1c4f084-bf3c-4108-9784-874f5ec8fc22" (UID: "b1c4f084-bf3c-4108-9784-874f5ec8fc22"). InnerVolumeSpecName "kube-api-access-x2dvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:20:05.650682 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:05.650656 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1c4f084-bf3c-4108-9784-874f5ec8fc22-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:05.650682 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:05.650679 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2dvc\" (UniqueName: \"kubernetes.io/projected/b1c4f084-bf3c-4108-9784-874f5ec8fc22-kube-api-access-x2dvc\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:05.650865 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:05.650690 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b1c4f084-bf3c-4108-9784-874f5ec8fc22-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:05.817498 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:05.817388 4289 generic.go:356] "Generic (PLEG): container finished" podID="b1c4f084-bf3c-4108-9784-874f5ec8fc22" containerID="687c8025f6dc0f1c631be0aaca4d96722200b15bd36e6559c5afc6d411c41dc0" exitCode=0 Oct 30 11:20:05.817498 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:05.817452 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6" Oct 30 11:20:05.817498 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:05.817474 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6" event={"ID":"b1c4f084-bf3c-4108-9784-874f5ec8fc22","Type":"ContainerDied","Data":"687c8025f6dc0f1c631be0aaca4d96722200b15bd36e6559c5afc6d411c41dc0"} Oct 30 11:20:05.817774 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:05.817543 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6" event={"ID":"b1c4f084-bf3c-4108-9784-874f5ec8fc22","Type":"ContainerDied","Data":"7c784cc5fa097ec739d7116b10ff3fee0ffe43fa218edf192f80161b39b2b7dc"} Oct 30 11:20:05.817774 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:05.817572 4289 scope.go:122] "RemoveContainer" containerID="687c8025f6dc0f1c631be0aaca4d96722200b15bd36e6559c5afc6d411c41dc0" Oct 30 11:20:05.825988 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:05.825968 4289 scope.go:122] "RemoveContainer" containerID="687c8025f6dc0f1c631be0aaca4d96722200b15bd36e6559c5afc6d411c41dc0" Oct 30 11:20:05.826275 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:20:05.826255 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"687c8025f6dc0f1c631be0aaca4d96722200b15bd36e6559c5afc6d411c41dc0\": container with ID starting with 687c8025f6dc0f1c631be0aaca4d96722200b15bd36e6559c5afc6d411c41dc0 not found: ID does not exist" containerID="687c8025f6dc0f1c631be0aaca4d96722200b15bd36e6559c5afc6d411c41dc0" Oct 30 11:20:05.826328 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:05.826282 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"687c8025f6dc0f1c631be0aaca4d96722200b15bd36e6559c5afc6d411c41dc0"} err="failed to get container status \"687c8025f6dc0f1c631be0aaca4d96722200b15bd36e6559c5afc6d411c41dc0\": rpc error: code = NotFound desc = could not find container \"687c8025f6dc0f1c631be0aaca4d96722200b15bd36e6559c5afc6d411c41dc0\": container with ID starting with 687c8025f6dc0f1c631be0aaca4d96722200b15bd36e6559c5afc6d411c41dc0 not found: ID does not exist" Oct 30 11:20:05.868663 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:05.868639 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6"] Oct 30 11:20:05.876136 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:05.876109 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-stxz6"] Oct 30 11:20:06.751362 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:06.751330 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:06.921554 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:06.921514 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:07.600740 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:07.600709 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:07.624785 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:07.624758 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1c4f084-bf3c-4108-9784-874f5ec8fc22" path="/var/lib/kubelet/pods/b1c4f084-bf3c-4108-9784-874f5ec8fc22/volumes" Oct 30 11:20:08.217655 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.217619 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:20:08.218065 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.217999 4289 cpu_manager.go:414] "containerMap: removing container" podUID="b1c4f084-bf3c-4108-9784-874f5ec8fc22" containerName="korrel8r" Oct 30 11:20:08.218065 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.218010 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1c4f084-bf3c-4108-9784-874f5ec8fc22" containerName="korrel8r" Oct 30 11:20:08.218065 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.218048 4289 cpu_manager.go:414] "containerMap: removing container" podUID="2dc7690a-2e29-4ca4-a520-8de5afa045f9" containerName="otc-container" Oct 30 11:20:08.218065 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.218053 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc7690a-2e29-4ca4-a520-8de5afa045f9" containerName="otc-container" Oct 30 11:20:08.218220 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.218112 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc7690a-2e29-4ca4-a520-8de5afa045f9" containerName="otc-container" Oct 30 11:20:08.218220 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.218119 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1c4f084-bf3c-4108-9784-874f5ec8fc22" containerName="korrel8r" Oct 30 11:20:08.222127 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.222112 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:20:08.248979 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.248954 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 11:20:08.249098 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.249005 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 11:20:08.260858 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.260830 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-e4cadd38\"" Oct 30 11:20:08.260976 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.260910 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-xn2fk\"" Oct 30 11:20:08.260976 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.260930 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 11:20:08.263706 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.263688 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:20:08.273532 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.273510 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7d1fe067-dea8-40d6-86ca-e19280132f32-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"7d1fe067-dea8-40d6-86ca-e19280132f32\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:20:08.273621 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.273569 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1fe067-dea8-40d6-86ca-e19280132f32-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7d1fe067-dea8-40d6-86ca-e19280132f32\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:20:08.273672 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.273660 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1fe067-dea8-40d6-86ca-e19280132f32-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7d1fe067-dea8-40d6-86ca-e19280132f32\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:20:08.273711 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.273685 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfrgk\" (UniqueName: \"kubernetes.io/projected/7d1fe067-dea8-40d6-86ca-e19280132f32-kube-api-access-mfrgk\") pod \"data-science-collector-collector-1\" (UID: \"7d1fe067-dea8-40d6-86ca-e19280132f32\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:20:08.273763 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.273708 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7d1fe067-dea8-40d6-86ca-e19280132f32-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7d1fe067-dea8-40d6-86ca-e19280132f32\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:20:08.374234 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.374199 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1fe067-dea8-40d6-86ca-e19280132f32-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7d1fe067-dea8-40d6-86ca-e19280132f32\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:20:08.374234 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.374234 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfrgk\" (UniqueName: \"kubernetes.io/projected/7d1fe067-dea8-40d6-86ca-e19280132f32-kube-api-access-mfrgk\") pod \"data-science-collector-collector-1\" (UID: \"7d1fe067-dea8-40d6-86ca-e19280132f32\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:20:08.374501 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.374256 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7d1fe067-dea8-40d6-86ca-e19280132f32-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7d1fe067-dea8-40d6-86ca-e19280132f32\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:20:08.374501 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.374288 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7d1fe067-dea8-40d6-86ca-e19280132f32-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"7d1fe067-dea8-40d6-86ca-e19280132f32\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:20:08.374501 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.374335 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1fe067-dea8-40d6-86ca-e19280132f32-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7d1fe067-dea8-40d6-86ca-e19280132f32\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:20:08.374923 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.374891 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7d1fe067-dea8-40d6-86ca-e19280132f32-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7d1fe067-dea8-40d6-86ca-e19280132f32\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:20:08.377035 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.377009 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1fe067-dea8-40d6-86ca-e19280132f32-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7d1fe067-dea8-40d6-86ca-e19280132f32\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:20:08.377035 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.377026 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1fe067-dea8-40d6-86ca-e19280132f32-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7d1fe067-dea8-40d6-86ca-e19280132f32\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:20:08.377190 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.377073 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7d1fe067-dea8-40d6-86ca-e19280132f32-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"7d1fe067-dea8-40d6-86ca-e19280132f32\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:20:08.397020 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.396991 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfrgk\" (UniqueName: \"kubernetes.io/projected/7d1fe067-dea8-40d6-86ca-e19280132f32-kube-api-access-mfrgk\") pod \"data-science-collector-collector-1\" (UID: \"7d1fe067-dea8-40d6-86ca-e19280132f32\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:20:08.534284 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.534259 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:20:08.832498 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.832384 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:08.833500 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:20:08.833445 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1fe067_dea8_40d6_86ca_e19280132f32.slice/crio-d8ee853870830d38ac78ba9dd5318339d0f4a80f27977b5069c991d126cfe7ea WatchSource:0}: Error finding container d8ee853870830d38ac78ba9dd5318339d0f4a80f27977b5069c991d126cfe7ea: Status 404 returned error can't find the container with id d8ee853870830d38ac78ba9dd5318339d0f4a80f27977b5069c991d126cfe7ea Oct 30 11:20:08.834571 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.834520 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:20:08.835962 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.835930 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:20:08.836049 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:08.835989 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:20:09.839690 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:09.839654 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7d1fe067-dea8-40d6-86ca-e19280132f32","Type":"ContainerStarted","Data":"516f37ee456e26c8caeadbfcd66ddef5a637ebe5964bff416c4061201b33a9fc"} Oct 30 11:20:09.840066 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:09.839717 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7d1fe067-dea8-40d6-86ca-e19280132f32","Type":"ContainerStarted","Data":"d8ee853870830d38ac78ba9dd5318339d0f4a80f27977b5069c991d126cfe7ea"} Oct 30 11:20:09.883735 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:09.883688 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.883673103 podStartE2EDuration="1.883673103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:20:09.881542854 +0000 UTC m=+2100.382922342" watchObservedRunningTime="2026-10-30 11:20:09.883673103 +0000 UTC m=+2100.385052587" Oct 30 11:20:11.014181 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.014148 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q"] Oct 30 11:20:11.014706 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.014421 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" podUID="3ba78eac-b3ef-4dc0-adf4-37897ca5b735" containerName="kube-rbac-proxy" containerID="cri-o://23bf9a9d69b72e89c854df1e58b2566695814d2d7d2016dcfd609a084cfe3ead" gracePeriod=30 Oct 30 11:20:11.043386 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.043360 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6"] Oct 30 11:20:11.043617 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.043596 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" podUID="8f4aed18-f28d-482c-a868-229cdadff6f3" containerName="kube-rbac-proxy" containerID="cri-o://e264a1beb7f6c23f7211feebc067b749a80445b1233013b4b0939b484625c48d" gracePeriod=30 Oct 30 11:20:11.043697 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.043611 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" podUID="8f4aed18-f28d-482c-a868-229cdadff6f3" containerName="prom-label-proxy" containerID="cri-o://f99bec7464c63f8187c41178beb393ff71d21d25a8cc1e5e6e7dac0204c1bde0" gracePeriod=30 Oct 30 11:20:11.043697 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.043614 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" podUID="8f4aed18-f28d-482c-a868-229cdadff6f3" containerName="method-gate" containerID="cri-o://550edb5e79119d463de7368eafed4d348a13f8ca5a0ac17abd7071a377faa3b6" gracePeriod=30 Oct 30 11:20:11.073317 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.073284 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw"] Oct 30 11:20:11.073545 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.073521 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" podUID="51c34556-aa72-4eed-ac1c-eb3f4534a62a" containerName="kube-rbac-proxy" containerID="cri-o://81319697799fa1982985be2d5b853401f685e90ba342616622202fa6aef473e7" gracePeriod=30 Oct 30 11:20:11.073600 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.073544 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" podUID="51c34556-aa72-4eed-ac1c-eb3f4534a62a" containerName="prom-label-proxy" containerID="cri-o://ea043eb61217ce74a94bebcbafac8c45b0133216a9d0a6d69c271b80be084358" gracePeriod=30 Oct 30 11:20:11.073658 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.073545 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" podUID="51c34556-aa72-4eed-ac1c-eb3f4534a62a" containerName="method-gate" containerID="cri-o://ca9b61f6b0fd9993abb1ed7c4a61294207e0e37731a02934fac96dac0ddf7a08" gracePeriod=30 Oct 30 11:20:11.106900 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.106872 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74"] Oct 30 11:20:11.107077 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.107056 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74" podUID="e473baa6-a55e-4bce-acb3-267ef3c8ad7e" containerName="korrel8r" containerID="cri-o://e483c93f8b697ada1bfd9997c401ecaf63a7859191aaa2124db6ab3ad707583f" gracePeriod=30 Oct 30 11:20:11.393169 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.389237 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" Oct 30 11:20:11.416059 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.416029 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" Oct 30 11:20:11.452058 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.451978 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74" Oct 30 11:20:11.455318 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.455290 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" Oct 30 11:20:11.501923 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.501877 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f4aed18-f28d-482c-a868-229cdadff6f3-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/8f4aed18-f28d-482c-a868-229cdadff6f3-prometheus-tls-ca\") pod \"8f4aed18-f28d-482c-a868-229cdadff6f3\" (UID: \"8f4aed18-f28d-482c-a868-229cdadff6f3\") " Oct 30 11:20:11.502129 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.501939 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-prometheus-ca\") pod \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\" (UID: \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\") " Oct 30 11:20:11.502129 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.501980 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-kube-api-access-n9pvq\" (UniqueName: \"kubernetes.io/projected/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-kube-api-access-n9pvq\") pod \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\" (UID: \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\") " Oct 30 11:20:11.502129 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.502012 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f4aed18-f28d-482c-a868-229cdadff6f3-tls\" (UniqueName: \"kubernetes.io/secret/8f4aed18-f28d-482c-a868-229cdadff6f3-tls\") pod \"8f4aed18-f28d-482c-a868-229cdadff6f3\" (UID: \"8f4aed18-f28d-482c-a868-229cdadff6f3\") " Oct 30 11:20:11.502129 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.502040 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-tls\" (UniqueName: \"kubernetes.io/secret/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-tls\") pod \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\" (UID: \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\") " Oct 30 11:20:11.502129 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.502068 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f4aed18-f28d-482c-a868-229cdadff6f3-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f4aed18-f28d-482c-a868-229cdadff6f3-client-ca\") pod \"8f4aed18-f28d-482c-a868-229cdadff6f3\" (UID: \"8f4aed18-f28d-482c-a868-229cdadff6f3\") " Oct 30 11:20:11.502129 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.502115 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-prometheus-client-cert\") pod \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\" (UID: \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\") " Oct 30 11:20:11.502588 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.502155 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e473baa6-a55e-4bce-acb3-267ef3c8ad7e-kube-api-access-gbtlb\" (UniqueName: \"kubernetes.io/projected/e473baa6-a55e-4bce-acb3-267ef3c8ad7e-kube-api-access-gbtlb\") pod \"e473baa6-a55e-4bce-acb3-267ef3c8ad7e\" (UID: \"e473baa6-a55e-4bce-acb3-267ef3c8ad7e\") " Oct 30 11:20:11.502588 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.502186 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f4aed18-f28d-482c-a868-229cdadff6f3-config\" (UniqueName: \"kubernetes.io/configmap/8f4aed18-f28d-482c-a868-229cdadff6f3-config\") pod \"8f4aed18-f28d-482c-a868-229cdadff6f3\" (UID: \"8f4aed18-f28d-482c-a868-229cdadff6f3\") " Oct 30 11:20:11.502588 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.502210 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f4aed18-f28d-482c-a868-229cdadff6f3-kube-api-access-dgvcx\" (UniqueName: \"kubernetes.io/projected/8f4aed18-f28d-482c-a868-229cdadff6f3-kube-api-access-dgvcx\") pod \"8f4aed18-f28d-482c-a868-229cdadff6f3\" (UID: \"8f4aed18-f28d-482c-a868-229cdadff6f3\") " Oct 30 11:20:11.502588 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.502282 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e473baa6-a55e-4bce-acb3-267ef3c8ad7e-tls\" (UniqueName: \"kubernetes.io/secret/e473baa6-a55e-4bce-acb3-267ef3c8ad7e-tls\") pod \"e473baa6-a55e-4bce-acb3-267ef3c8ad7e\" (UID: \"e473baa6-a55e-4bce-acb3-267ef3c8ad7e\") " Oct 30 11:20:11.502588 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.502305 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-kube-rbac-proxy-config\") pod \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\" (UID: \"3ba78eac-b3ef-4dc0-adf4-37897ca5b735\") " Oct 30 11:20:11.502588 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.502327 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e473baa6-a55e-4bce-acb3-267ef3c8ad7e-config\" (UniqueName: \"kubernetes.io/configmap/e473baa6-a55e-4bce-acb3-267ef3c8ad7e-config\") pod \"e473baa6-a55e-4bce-acb3-267ef3c8ad7e\" (UID: \"e473baa6-a55e-4bce-acb3-267ef3c8ad7e\") " Oct 30 11:20:11.502588 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.502323 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4aed18-f28d-482c-a868-229cdadff6f3-prometheus-tls-ca" pod "8f4aed18-f28d-482c-a868-229cdadff6f3" (UID: "8f4aed18-f28d-482c-a868-229cdadff6f3"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:20:11.502588 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.502541 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/8f4aed18-f28d-482c-a868-229cdadff6f3-prometheus-tls-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:11.502995 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.502896 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e473baa6-a55e-4bce-acb3-267ef3c8ad7e-config" pod "e473baa6-a55e-4bce-acb3-267ef3c8ad7e" (UID: "e473baa6-a55e-4bce-acb3-267ef3c8ad7e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:20:11.503465 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.503397 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4aed18-f28d-482c-a868-229cdadff6f3-client-ca" pod "8f4aed18-f28d-482c-a868-229cdadff6f3" (UID: "8f4aed18-f28d-482c-a868-229cdadff6f3"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:20:11.503917 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.503879 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4aed18-f28d-482c-a868-229cdadff6f3-config" pod "8f4aed18-f28d-482c-a868-229cdadff6f3" (UID: "8f4aed18-f28d-482c-a868-229cdadff6f3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:20:11.504348 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.504323 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-prometheus-ca" pod "3ba78eac-b3ef-4dc0-adf4-37897ca5b735" (UID: "3ba78eac-b3ef-4dc0-adf4-37897ca5b735"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:20:11.506375 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.506354 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-kube-rbac-proxy-config" pod "3ba78eac-b3ef-4dc0-adf4-37897ca5b735" (UID: "3ba78eac-b3ef-4dc0-adf4-37897ca5b735"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:20:11.507767 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.507732 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e473baa6-a55e-4bce-acb3-267ef3c8ad7e-tls" pod "e473baa6-a55e-4bce-acb3-267ef3c8ad7e" (UID: "e473baa6-a55e-4bce-acb3-267ef3c8ad7e"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:20:11.508021 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.507986 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-prometheus-client-cert" pod "3ba78eac-b3ef-4dc0-adf4-37897ca5b735" (UID: "3ba78eac-b3ef-4dc0-adf4-37897ca5b735"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:20:11.508021 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.508006 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-kube-api-access-n9pvq" pod "3ba78eac-b3ef-4dc0-adf4-37897ca5b735" (UID: "3ba78eac-b3ef-4dc0-adf4-37897ca5b735"). InnerVolumeSpecName "kube-api-access-n9pvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:20:11.508190 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.508023 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f4aed18-f28d-482c-a868-229cdadff6f3-tls" pod "8f4aed18-f28d-482c-a868-229cdadff6f3" (UID: "8f4aed18-f28d-482c-a868-229cdadff6f3"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:20:11.508483 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.508456 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-tls" pod "3ba78eac-b3ef-4dc0-adf4-37897ca5b735" (UID: "3ba78eac-b3ef-4dc0-adf4-37897ca5b735"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:20:11.509138 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.509111 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f4aed18-f28d-482c-a868-229cdadff6f3-kube-api-access-dgvcx" pod "8f4aed18-f28d-482c-a868-229cdadff6f3" (UID: "8f4aed18-f28d-482c-a868-229cdadff6f3"). InnerVolumeSpecName "kube-api-access-dgvcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:20:11.509968 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.509943 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e473baa6-a55e-4bce-acb3-267ef3c8ad7e-kube-api-access-gbtlb" pod "e473baa6-a55e-4bce-acb3-267ef3c8ad7e" (UID: "e473baa6-a55e-4bce-acb3-267ef3c8ad7e"). InnerVolumeSpecName "kube-api-access-gbtlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:20:11.552822 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.552665 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:11.603374 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.603322 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51c34556-aa72-4eed-ac1c-eb3f4534a62a-kube-api-access-6nwdj\" (UniqueName: \"kubernetes.io/projected/51c34556-aa72-4eed-ac1c-eb3f4534a62a-kube-api-access-6nwdj\") pod \"51c34556-aa72-4eed-ac1c-eb3f4534a62a\" (UID: \"51c34556-aa72-4eed-ac1c-eb3f4534a62a\") " Oct 30 11:20:11.603575 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.603388 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51c34556-aa72-4eed-ac1c-eb3f4534a62a-config\" (UniqueName: \"kubernetes.io/configmap/51c34556-aa72-4eed-ac1c-eb3f4534a62a-config\") pod \"51c34556-aa72-4eed-ac1c-eb3f4534a62a\" (UID: \"51c34556-aa72-4eed-ac1c-eb3f4534a62a\") " Oct 30 11:20:11.603575 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.603420 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51c34556-aa72-4eed-ac1c-eb3f4534a62a-tls\" (UniqueName: \"kubernetes.io/secret/51c34556-aa72-4eed-ac1c-eb3f4534a62a-tls\") pod \"51c34556-aa72-4eed-ac1c-eb3f4534a62a\" (UID: \"51c34556-aa72-4eed-ac1c-eb3f4534a62a\") " Oct 30 11:20:11.604025 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.603851 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbtlb\" (UniqueName: \"kubernetes.io/projected/e473baa6-a55e-4bce-acb3-267ef3c8ad7e-kube-api-access-gbtlb\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:11.604025 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.603862 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51c34556-aa72-4eed-ac1c-eb3f4534a62a-config" pod "51c34556-aa72-4eed-ac1c-eb3f4534a62a" (UID: "51c34556-aa72-4eed-ac1c-eb3f4534a62a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:20:11.604025 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.603878 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f4aed18-f28d-482c-a868-229cdadff6f3-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:11.604025 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.603893 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgvcx\" (UniqueName: \"kubernetes.io/projected/8f4aed18-f28d-482c-a868-229cdadff6f3-kube-api-access-dgvcx\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:11.604273 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.604197 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e473baa6-a55e-4bce-acb3-267ef3c8ad7e-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:11.604273 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.604222 4289 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-kube-rbac-proxy-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:11.604273 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.604236 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e473baa6-a55e-4bce-acb3-267ef3c8ad7e-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:11.604273 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.604250 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-prometheus-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:11.604273 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.604263 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9pvq\" (UniqueName: \"kubernetes.io/projected/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-kube-api-access-n9pvq\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:11.604798 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.604275 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8f4aed18-f28d-482c-a868-229cdadff6f3-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:11.604798 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.604287 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:11.604798 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.604299 4289 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/8f4aed18-f28d-482c-a868-229cdadff6f3-client-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:11.604798 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.604311 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/3ba78eac-b3ef-4dc0-adf4-37897ca5b735-prometheus-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:11.609153 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.606045 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51c34556-aa72-4eed-ac1c-eb3f4534a62a-tls" pod "51c34556-aa72-4eed-ac1c-eb3f4534a62a" (UID: "51c34556-aa72-4eed-ac1c-eb3f4534a62a"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:20:11.609153 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.606698 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c34556-aa72-4eed-ac1c-eb3f4534a62a-kube-api-access-6nwdj" pod "51c34556-aa72-4eed-ac1c-eb3f4534a62a" (UID: "51c34556-aa72-4eed-ac1c-eb3f4534a62a"). InnerVolumeSpecName "kube-api-access-6nwdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:20:11.610384 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.610360 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:20:11.610905 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.610731 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="3ff8a282-f987-4c65-911d-b56e9ea5dd09" containerName="config-reloader" containerID="cri-o://3c74ee91d769889d58aba84262f48d31c80ce163392f6721505a6fa6d6a2801f" gracePeriod=600 Oct 30 11:20:11.610905 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.610740 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="3ff8a282-f987-4c65-911d-b56e9ea5dd09" containerName="thanos-sidecar" containerID="cri-o://deb60fcd295d51457107e09e69000da518ab9300141e985a5a0b26ebae061751" gracePeriod=600 Oct 30 11:20:11.610905 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.610708 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="3ff8a282-f987-4c65-911d-b56e9ea5dd09" containerName="prometheus" containerID="cri-o://1420776adc26b76c076fcb26b0435d0e8892bcfb9a12dc85939a3b6ff762a564" gracePeriod=600 Oct 30 11:20:11.619088 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.619060 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 11:20:11.619514 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.619482 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d" containerName="alertmanager" containerID="cri-o://446d7d8d01226fc914207861ccaf24083c827da98d224fbe74ca6bcb4689c130" gracePeriod=120 Oct 30 11:20:11.619689 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.619667 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d" containerName="config-reloader" containerID="cri-o://288302b0523fb8b3fc3761bf69b697aeda8bfb9c9b17c819cf5bc439fc2a6fce" gracePeriod=120 Oct 30 11:20:11.705129 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.705099 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nwdj\" (UniqueName: \"kubernetes.io/projected/51c34556-aa72-4eed-ac1c-eb3f4534a62a-kube-api-access-6nwdj\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:11.705129 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.705126 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/51c34556-aa72-4eed-ac1c-eb3f4534a62a-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:11.705129 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.705136 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/51c34556-aa72-4eed-ac1c-eb3f4534a62a-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:11.733610 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.733578 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:11.850862 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.850827 4289 generic.go:356] "Generic (PLEG): container finished" podID="8f4aed18-f28d-482c-a868-229cdadff6f3" containerID="f99bec7464c63f8187c41178beb393ff71d21d25a8cc1e5e6e7dac0204c1bde0" exitCode=0 Oct 30 11:20:11.850862 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.850857 4289 generic.go:356] "Generic (PLEG): container finished" podID="8f4aed18-f28d-482c-a868-229cdadff6f3" containerID="550edb5e79119d463de7368eafed4d348a13f8ca5a0ac17abd7071a377faa3b6" exitCode=0 Oct 30 11:20:11.850862 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.850865 4289 generic.go:356] "Generic (PLEG): container finished" podID="8f4aed18-f28d-482c-a868-229cdadff6f3" containerID="e264a1beb7f6c23f7211feebc067b749a80445b1233013b4b0939b484625c48d" exitCode=0 Oct 30 11:20:11.851124 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.850891 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" Oct 30 11:20:11.851124 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.850902 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" event={"ID":"8f4aed18-f28d-482c-a868-229cdadff6f3","Type":"ContainerDied","Data":"f99bec7464c63f8187c41178beb393ff71d21d25a8cc1e5e6e7dac0204c1bde0"} Oct 30 11:20:11.851124 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.850981 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" event={"ID":"8f4aed18-f28d-482c-a868-229cdadff6f3","Type":"ContainerDied","Data":"550edb5e79119d463de7368eafed4d348a13f8ca5a0ac17abd7071a377faa3b6"} Oct 30 11:20:11.851124 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.851021 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" event={"ID":"8f4aed18-f28d-482c-a868-229cdadff6f3","Type":"ContainerDied","Data":"e264a1beb7f6c23f7211feebc067b749a80445b1233013b4b0939b484625c48d"} Oct 30 11:20:11.851124 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.851051 4289 scope.go:122] "RemoveContainer" containerID="f99bec7464c63f8187c41178beb393ff71d21d25a8cc1e5e6e7dac0204c1bde0" Oct 30 11:20:11.851361 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.851058 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6" event={"ID":"8f4aed18-f28d-482c-a868-229cdadff6f3","Type":"ContainerDied","Data":"e34f5b694fad5ad3cef02a9ce0a7c6e8054646254dd9324113f4fcd5a588b1b0"} Oct 30 11:20:11.853250 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.853111 4289 generic.go:356] "Generic (PLEG): container finished" podID="51c34556-aa72-4eed-ac1c-eb3f4534a62a" containerID="ea043eb61217ce74a94bebcbafac8c45b0133216a9d0a6d69c271b80be084358" exitCode=0 Oct 30 11:20:11.853250 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.853138 4289 generic.go:356] "Generic (PLEG): container finished" podID="51c34556-aa72-4eed-ac1c-eb3f4534a62a" containerID="ca9b61f6b0fd9993abb1ed7c4a61294207e0e37731a02934fac96dac0ddf7a08" exitCode=0 Oct 30 11:20:11.853250 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.853148 4289 generic.go:356] "Generic (PLEG): container finished" podID="51c34556-aa72-4eed-ac1c-eb3f4534a62a" containerID="81319697799fa1982985be2d5b853401f685e90ba342616622202fa6aef473e7" exitCode=0 Oct 30 11:20:11.853250 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.853175 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" Oct 30 11:20:11.853250 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.853197 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" event={"ID":"51c34556-aa72-4eed-ac1c-eb3f4534a62a","Type":"ContainerDied","Data":"ea043eb61217ce74a94bebcbafac8c45b0133216a9d0a6d69c271b80be084358"} Oct 30 11:20:11.853621 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.853273 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" event={"ID":"51c34556-aa72-4eed-ac1c-eb3f4534a62a","Type":"ContainerDied","Data":"ca9b61f6b0fd9993abb1ed7c4a61294207e0e37731a02934fac96dac0ddf7a08"} Oct 30 11:20:11.853621 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.853322 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" event={"ID":"51c34556-aa72-4eed-ac1c-eb3f4534a62a","Type":"ContainerDied","Data":"81319697799fa1982985be2d5b853401f685e90ba342616622202fa6aef473e7"} Oct 30 11:20:11.853621 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.853361 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw" event={"ID":"51c34556-aa72-4eed-ac1c-eb3f4534a62a","Type":"ContainerDied","Data":"736d136facbfe0a82d8b870effea1528db0b17aad68769f638a48f4572095107"} Oct 30 11:20:11.855075 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.854932 4289 generic.go:356] "Generic (PLEG): container finished" podID="e473baa6-a55e-4bce-acb3-267ef3c8ad7e" containerID="e483c93f8b697ada1bfd9997c401ecaf63a7859191aaa2124db6ab3ad707583f" exitCode=0 Oct 30 11:20:11.855075 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.854960 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74" event={"ID":"e473baa6-a55e-4bce-acb3-267ef3c8ad7e","Type":"ContainerDied","Data":"e483c93f8b697ada1bfd9997c401ecaf63a7859191aaa2124db6ab3ad707583f"} Oct 30 11:20:11.855075 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.854980 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74" Oct 30 11:20:11.855075 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.855023 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74" event={"ID":"e473baa6-a55e-4bce-acb3-267ef3c8ad7e","Type":"ContainerDied","Data":"42d2addc01fd52657cecb7a1b1d9f3090a062c59b0996615257c86fca626dc5b"} Oct 30 11:20:11.858087 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.858065 4289 generic.go:356] "Generic (PLEG): container finished" podID="3ff8a282-f987-4c65-911d-b56e9ea5dd09" containerID="deb60fcd295d51457107e09e69000da518ab9300141e985a5a0b26ebae061751" exitCode=0 Oct 30 11:20:11.858167 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.858088 4289 generic.go:356] "Generic (PLEG): container finished" podID="3ff8a282-f987-4c65-911d-b56e9ea5dd09" containerID="1420776adc26b76c076fcb26b0435d0e8892bcfb9a12dc85939a3b6ff762a564" exitCode=0 Oct 30 11:20:11.858167 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.858131 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"3ff8a282-f987-4c65-911d-b56e9ea5dd09","Type":"ContainerDied","Data":"deb60fcd295d51457107e09e69000da518ab9300141e985a5a0b26ebae061751"} Oct 30 11:20:11.858249 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.858179 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"3ff8a282-f987-4c65-911d-b56e9ea5dd09","Type":"ContainerDied","Data":"1420776adc26b76c076fcb26b0435d0e8892bcfb9a12dc85939a3b6ff762a564"} Oct 30 11:20:11.860475 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.860454 4289 generic.go:356] "Generic (PLEG): container finished" podID="39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d" containerID="288302b0523fb8b3fc3761bf69b697aeda8bfb9c9b17c819cf5bc439fc2a6fce" exitCode=0 Oct 30 11:20:11.860575 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.860523 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d","Type":"ContainerDied","Data":"288302b0523fb8b3fc3761bf69b697aeda8bfb9c9b17c819cf5bc439fc2a6fce"} Oct 30 11:20:11.861151 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.861126 4289 scope.go:122] "RemoveContainer" containerID="550edb5e79119d463de7368eafed4d348a13f8ca5a0ac17abd7071a377faa3b6" Oct 30 11:20:11.861717 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.861698 4289 generic.go:356] "Generic (PLEG): container finished" podID="3ba78eac-b3ef-4dc0-adf4-37897ca5b735" containerID="23bf9a9d69b72e89c854df1e58b2566695814d2d7d2016dcfd609a084cfe3ead" exitCode=0 Oct 30 11:20:11.861800 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.861733 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" event={"ID":"3ba78eac-b3ef-4dc0-adf4-37897ca5b735","Type":"ContainerDied","Data":"23bf9a9d69b72e89c854df1e58b2566695814d2d7d2016dcfd609a084cfe3ead"} Oct 30 11:20:11.861800 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.861740 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" Oct 30 11:20:11.861800 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.861767 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q" event={"ID":"3ba78eac-b3ef-4dc0-adf4-37897ca5b735","Type":"ContainerDied","Data":"a69195966b7eabda9ab0f859e0171fdf468e452dddf99e634fbe948f8981afe6"} Oct 30 11:20:11.868516 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.868500 4289 scope.go:122] "RemoveContainer" containerID="e264a1beb7f6c23f7211feebc067b749a80445b1233013b4b0939b484625c48d" Oct 30 11:20:11.875273 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.875251 4289 scope.go:122] "RemoveContainer" containerID="f99bec7464c63f8187c41178beb393ff71d21d25a8cc1e5e6e7dac0204c1bde0" Oct 30 11:20:11.875560 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:20:11.875543 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f99bec7464c63f8187c41178beb393ff71d21d25a8cc1e5e6e7dac0204c1bde0\": container with ID starting with f99bec7464c63f8187c41178beb393ff71d21d25a8cc1e5e6e7dac0204c1bde0 not found: ID does not exist" containerID="f99bec7464c63f8187c41178beb393ff71d21d25a8cc1e5e6e7dac0204c1bde0" Oct 30 11:20:11.875648 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.875572 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f99bec7464c63f8187c41178beb393ff71d21d25a8cc1e5e6e7dac0204c1bde0"} err="failed to get container status \"f99bec7464c63f8187c41178beb393ff71d21d25a8cc1e5e6e7dac0204c1bde0\": rpc error: code = NotFound desc = could not find container \"f99bec7464c63f8187c41178beb393ff71d21d25a8cc1e5e6e7dac0204c1bde0\": container with ID starting with f99bec7464c63f8187c41178beb393ff71d21d25a8cc1e5e6e7dac0204c1bde0 not found: ID does not exist" Oct 30 11:20:11.875648 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.875594 4289 scope.go:122] "RemoveContainer" containerID="550edb5e79119d463de7368eafed4d348a13f8ca5a0ac17abd7071a377faa3b6" Oct 30 11:20:11.875839 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:20:11.875815 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"550edb5e79119d463de7368eafed4d348a13f8ca5a0ac17abd7071a377faa3b6\": container with ID starting with 550edb5e79119d463de7368eafed4d348a13f8ca5a0ac17abd7071a377faa3b6 not found: ID does not exist" containerID="550edb5e79119d463de7368eafed4d348a13f8ca5a0ac17abd7071a377faa3b6" Oct 30 11:20:11.875913 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.875847 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"550edb5e79119d463de7368eafed4d348a13f8ca5a0ac17abd7071a377faa3b6"} err="failed to get container status \"550edb5e79119d463de7368eafed4d348a13f8ca5a0ac17abd7071a377faa3b6\": rpc error: code = NotFound desc = could not find container \"550edb5e79119d463de7368eafed4d348a13f8ca5a0ac17abd7071a377faa3b6\": container with ID starting with 550edb5e79119d463de7368eafed4d348a13f8ca5a0ac17abd7071a377faa3b6 not found: ID does not exist" Oct 30 11:20:11.875913 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.875867 4289 scope.go:122] "RemoveContainer" containerID="e264a1beb7f6c23f7211feebc067b749a80445b1233013b4b0939b484625c48d" Oct 30 11:20:11.876131 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:20:11.876114 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e264a1beb7f6c23f7211feebc067b749a80445b1233013b4b0939b484625c48d\": container with ID starting with e264a1beb7f6c23f7211feebc067b749a80445b1233013b4b0939b484625c48d not found: ID does not exist" containerID="e264a1beb7f6c23f7211feebc067b749a80445b1233013b4b0939b484625c48d" Oct 30 11:20:11.876169 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.876140 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e264a1beb7f6c23f7211feebc067b749a80445b1233013b4b0939b484625c48d"} err="failed to get container status \"e264a1beb7f6c23f7211feebc067b749a80445b1233013b4b0939b484625c48d\": rpc error: code = NotFound desc = could not find container \"e264a1beb7f6c23f7211feebc067b749a80445b1233013b4b0939b484625c48d\": container with ID starting with e264a1beb7f6c23f7211feebc067b749a80445b1233013b4b0939b484625c48d not found: ID does not exist" Oct 30 11:20:11.876169 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.876158 4289 scope.go:122] "RemoveContainer" containerID="f99bec7464c63f8187c41178beb393ff71d21d25a8cc1e5e6e7dac0204c1bde0" Oct 30 11:20:11.876378 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.876356 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f99bec7464c63f8187c41178beb393ff71d21d25a8cc1e5e6e7dac0204c1bde0"} err="failed to get container status \"f99bec7464c63f8187c41178beb393ff71d21d25a8cc1e5e6e7dac0204c1bde0\": rpc error: code = NotFound desc = could not find container \"f99bec7464c63f8187c41178beb393ff71d21d25a8cc1e5e6e7dac0204c1bde0\": container with ID starting with f99bec7464c63f8187c41178beb393ff71d21d25a8cc1e5e6e7dac0204c1bde0 not found: ID does not exist" Oct 30 11:20:11.876378 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.876376 4289 scope.go:122] "RemoveContainer" containerID="550edb5e79119d463de7368eafed4d348a13f8ca5a0ac17abd7071a377faa3b6" Oct 30 11:20:11.876597 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.876577 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"550edb5e79119d463de7368eafed4d348a13f8ca5a0ac17abd7071a377faa3b6"} err="failed to get container status \"550edb5e79119d463de7368eafed4d348a13f8ca5a0ac17abd7071a377faa3b6\": rpc error: code = NotFound desc = could not find container \"550edb5e79119d463de7368eafed4d348a13f8ca5a0ac17abd7071a377faa3b6\": container with ID starting with 550edb5e79119d463de7368eafed4d348a13f8ca5a0ac17abd7071a377faa3b6 not found: ID does not exist" Oct 30 11:20:11.876597 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.876596 4289 scope.go:122] "RemoveContainer" containerID="e264a1beb7f6c23f7211feebc067b749a80445b1233013b4b0939b484625c48d" Oct 30 11:20:11.876806 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.876790 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e264a1beb7f6c23f7211feebc067b749a80445b1233013b4b0939b484625c48d"} err="failed to get container status \"e264a1beb7f6c23f7211feebc067b749a80445b1233013b4b0939b484625c48d\": rpc error: code = NotFound desc = could not find container \"e264a1beb7f6c23f7211feebc067b749a80445b1233013b4b0939b484625c48d\": container with ID starting with e264a1beb7f6c23f7211feebc067b749a80445b1233013b4b0939b484625c48d not found: ID does not exist" Oct 30 11:20:11.876855 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.876806 4289 scope.go:122] "RemoveContainer" containerID="f99bec7464c63f8187c41178beb393ff71d21d25a8cc1e5e6e7dac0204c1bde0" Oct 30 11:20:11.877057 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.877038 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f99bec7464c63f8187c41178beb393ff71d21d25a8cc1e5e6e7dac0204c1bde0"} err="failed to get container status \"f99bec7464c63f8187c41178beb393ff71d21d25a8cc1e5e6e7dac0204c1bde0\": rpc error: code = NotFound desc = could not find container \"f99bec7464c63f8187c41178beb393ff71d21d25a8cc1e5e6e7dac0204c1bde0\": container with ID starting with f99bec7464c63f8187c41178beb393ff71d21d25a8cc1e5e6e7dac0204c1bde0 not found: ID does not exist" Oct 30 11:20:11.877057 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.877057 4289 scope.go:122] "RemoveContainer" containerID="550edb5e79119d463de7368eafed4d348a13f8ca5a0ac17abd7071a377faa3b6" Oct 30 11:20:11.877377 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.877349 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"550edb5e79119d463de7368eafed4d348a13f8ca5a0ac17abd7071a377faa3b6"} err="failed to get container status \"550edb5e79119d463de7368eafed4d348a13f8ca5a0ac17abd7071a377faa3b6\": rpc error: code = NotFound desc = could not find container \"550edb5e79119d463de7368eafed4d348a13f8ca5a0ac17abd7071a377faa3b6\": container with ID starting with 550edb5e79119d463de7368eafed4d348a13f8ca5a0ac17abd7071a377faa3b6 not found: ID does not exist" Oct 30 11:20:11.877377 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.877377 4289 scope.go:122] "RemoveContainer" containerID="e264a1beb7f6c23f7211feebc067b749a80445b1233013b4b0939b484625c48d" Oct 30 11:20:11.877747 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.877723 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e264a1beb7f6c23f7211feebc067b749a80445b1233013b4b0939b484625c48d"} err="failed to get container status \"e264a1beb7f6c23f7211feebc067b749a80445b1233013b4b0939b484625c48d\": rpc error: code = NotFound desc = could not find container \"e264a1beb7f6c23f7211feebc067b749a80445b1233013b4b0939b484625c48d\": container with ID starting with e264a1beb7f6c23f7211feebc067b749a80445b1233013b4b0939b484625c48d not found: ID does not exist" Oct 30 11:20:11.877854 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.877748 4289 scope.go:122] "RemoveContainer" containerID="ea043eb61217ce74a94bebcbafac8c45b0133216a9d0a6d69c271b80be084358" Oct 30 11:20:11.886669 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.886645 4289 scope.go:122] "RemoveContainer" containerID="ca9b61f6b0fd9993abb1ed7c4a61294207e0e37731a02934fac96dac0ddf7a08" Oct 30 11:20:11.908168 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.908142 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q"] Oct 30 11:20:11.913907 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.913888 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-krm2q"] Oct 30 11:20:11.939329 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.939308 4289 scope.go:122] "RemoveContainer" containerID="81319697799fa1982985be2d5b853401f685e90ba342616622202fa6aef473e7" Oct 30 11:20:11.948123 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.948102 4289 scope.go:122] "RemoveContainer" containerID="ea043eb61217ce74a94bebcbafac8c45b0133216a9d0a6d69c271b80be084358" Oct 30 11:20:11.948512 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:20:11.948485 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea043eb61217ce74a94bebcbafac8c45b0133216a9d0a6d69c271b80be084358\": container with ID starting with ea043eb61217ce74a94bebcbafac8c45b0133216a9d0a6d69c271b80be084358 not found: ID does not exist" containerID="ea043eb61217ce74a94bebcbafac8c45b0133216a9d0a6d69c271b80be084358" Oct 30 11:20:11.948607 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.948523 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea043eb61217ce74a94bebcbafac8c45b0133216a9d0a6d69c271b80be084358"} err="failed to get container status \"ea043eb61217ce74a94bebcbafac8c45b0133216a9d0a6d69c271b80be084358\": rpc error: code = NotFound desc = could not find container \"ea043eb61217ce74a94bebcbafac8c45b0133216a9d0a6d69c271b80be084358\": container with ID starting with ea043eb61217ce74a94bebcbafac8c45b0133216a9d0a6d69c271b80be084358 not found: ID does not exist" Oct 30 11:20:11.948607 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.948547 4289 scope.go:122] "RemoveContainer" containerID="ca9b61f6b0fd9993abb1ed7c4a61294207e0e37731a02934fac96dac0ddf7a08" Oct 30 11:20:11.948933 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:20:11.948903 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca9b61f6b0fd9993abb1ed7c4a61294207e0e37731a02934fac96dac0ddf7a08\": container with ID starting with ca9b61f6b0fd9993abb1ed7c4a61294207e0e37731a02934fac96dac0ddf7a08 not found: ID does not exist" containerID="ca9b61f6b0fd9993abb1ed7c4a61294207e0e37731a02934fac96dac0ddf7a08" Oct 30 11:20:11.948979 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.948948 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca9b61f6b0fd9993abb1ed7c4a61294207e0e37731a02934fac96dac0ddf7a08"} err="failed to get container status \"ca9b61f6b0fd9993abb1ed7c4a61294207e0e37731a02934fac96dac0ddf7a08\": rpc error: code = NotFound desc = could not find container \"ca9b61f6b0fd9993abb1ed7c4a61294207e0e37731a02934fac96dac0ddf7a08\": container with ID starting with ca9b61f6b0fd9993abb1ed7c4a61294207e0e37731a02934fac96dac0ddf7a08 not found: ID does not exist" Oct 30 11:20:11.949017 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.948977 4289 scope.go:122] "RemoveContainer" containerID="81319697799fa1982985be2d5b853401f685e90ba342616622202fa6aef473e7" Oct 30 11:20:11.949322 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:20:11.949299 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81319697799fa1982985be2d5b853401f685e90ba342616622202fa6aef473e7\": container with ID starting with 81319697799fa1982985be2d5b853401f685e90ba342616622202fa6aef473e7 not found: ID does not exist" containerID="81319697799fa1982985be2d5b853401f685e90ba342616622202fa6aef473e7" Oct 30 11:20:11.949404 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.949332 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81319697799fa1982985be2d5b853401f685e90ba342616622202fa6aef473e7"} err="failed to get container status \"81319697799fa1982985be2d5b853401f685e90ba342616622202fa6aef473e7\": rpc error: code = NotFound desc = could not find container \"81319697799fa1982985be2d5b853401f685e90ba342616622202fa6aef473e7\": container with ID starting with 81319697799fa1982985be2d5b853401f685e90ba342616622202fa6aef473e7 not found: ID does not exist" Oct 30 11:20:11.949404 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.949348 4289 scope.go:122] "RemoveContainer" containerID="ea043eb61217ce74a94bebcbafac8c45b0133216a9d0a6d69c271b80be084358" Oct 30 11:20:11.949940 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.949904 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea043eb61217ce74a94bebcbafac8c45b0133216a9d0a6d69c271b80be084358"} err="failed to get container status \"ea043eb61217ce74a94bebcbafac8c45b0133216a9d0a6d69c271b80be084358\": rpc error: code = NotFound desc = could not find container \"ea043eb61217ce74a94bebcbafac8c45b0133216a9d0a6d69c271b80be084358\": container with ID starting with ea043eb61217ce74a94bebcbafac8c45b0133216a9d0a6d69c271b80be084358 not found: ID does not exist" Oct 30 11:20:11.949940 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.949935 4289 scope.go:122] "RemoveContainer" containerID="ca9b61f6b0fd9993abb1ed7c4a61294207e0e37731a02934fac96dac0ddf7a08" Oct 30 11:20:11.950212 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.950192 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca9b61f6b0fd9993abb1ed7c4a61294207e0e37731a02934fac96dac0ddf7a08"} err="failed to get container status \"ca9b61f6b0fd9993abb1ed7c4a61294207e0e37731a02934fac96dac0ddf7a08\": rpc error: code = NotFound desc = could not find container \"ca9b61f6b0fd9993abb1ed7c4a61294207e0e37731a02934fac96dac0ddf7a08\": container with ID starting with ca9b61f6b0fd9993abb1ed7c4a61294207e0e37731a02934fac96dac0ddf7a08 not found: ID does not exist" Oct 30 11:20:11.950267 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.950214 4289 scope.go:122] "RemoveContainer" containerID="81319697799fa1982985be2d5b853401f685e90ba342616622202fa6aef473e7" Oct 30 11:20:11.950482 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.950460 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81319697799fa1982985be2d5b853401f685e90ba342616622202fa6aef473e7"} err="failed to get container status \"81319697799fa1982985be2d5b853401f685e90ba342616622202fa6aef473e7\": rpc error: code = NotFound desc = could not find container \"81319697799fa1982985be2d5b853401f685e90ba342616622202fa6aef473e7\": container with ID starting with 81319697799fa1982985be2d5b853401f685e90ba342616622202fa6aef473e7 not found: ID does not exist" Oct 30 11:20:11.950546 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.950483 4289 scope.go:122] "RemoveContainer" containerID="ea043eb61217ce74a94bebcbafac8c45b0133216a9d0a6d69c271b80be084358" Oct 30 11:20:11.950726 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.950705 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea043eb61217ce74a94bebcbafac8c45b0133216a9d0a6d69c271b80be084358"} err="failed to get container status \"ea043eb61217ce74a94bebcbafac8c45b0133216a9d0a6d69c271b80be084358\": rpc error: code = NotFound desc = could not find container \"ea043eb61217ce74a94bebcbafac8c45b0133216a9d0a6d69c271b80be084358\": container with ID starting with ea043eb61217ce74a94bebcbafac8c45b0133216a9d0a6d69c271b80be084358 not found: ID does not exist" Oct 30 11:20:11.950774 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.950728 4289 scope.go:122] "RemoveContainer" containerID="ca9b61f6b0fd9993abb1ed7c4a61294207e0e37731a02934fac96dac0ddf7a08" Oct 30 11:20:11.951015 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.950992 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca9b61f6b0fd9993abb1ed7c4a61294207e0e37731a02934fac96dac0ddf7a08"} err="failed to get container status \"ca9b61f6b0fd9993abb1ed7c4a61294207e0e37731a02934fac96dac0ddf7a08\": rpc error: code = NotFound desc = could not find container \"ca9b61f6b0fd9993abb1ed7c4a61294207e0e37731a02934fac96dac0ddf7a08\": container with ID starting with ca9b61f6b0fd9993abb1ed7c4a61294207e0e37731a02934fac96dac0ddf7a08 not found: ID does not exist" Oct 30 11:20:11.951067 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.951017 4289 scope.go:122] "RemoveContainer" containerID="81319697799fa1982985be2d5b853401f685e90ba342616622202fa6aef473e7" Oct 30 11:20:11.951288 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.951271 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81319697799fa1982985be2d5b853401f685e90ba342616622202fa6aef473e7"} err="failed to get container status \"81319697799fa1982985be2d5b853401f685e90ba342616622202fa6aef473e7\": rpc error: code = NotFound desc = could not find container \"81319697799fa1982985be2d5b853401f685e90ba342616622202fa6aef473e7\": container with ID starting with 81319697799fa1982985be2d5b853401f685e90ba342616622202fa6aef473e7 not found: ID does not exist" Oct 30 11:20:11.951350 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.951289 4289 scope.go:122] "RemoveContainer" containerID="e483c93f8b697ada1bfd9997c401ecaf63a7859191aaa2124db6ab3ad707583f" Oct 30 11:20:11.964519 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.964500 4289 scope.go:122] "RemoveContainer" containerID="e483c93f8b697ada1bfd9997c401ecaf63a7859191aaa2124db6ab3ad707583f" Oct 30 11:20:11.964977 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:20:11.964851 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e483c93f8b697ada1bfd9997c401ecaf63a7859191aaa2124db6ab3ad707583f\": container with ID starting with e483c93f8b697ada1bfd9997c401ecaf63a7859191aaa2124db6ab3ad707583f not found: ID does not exist" containerID="e483c93f8b697ada1bfd9997c401ecaf63a7859191aaa2124db6ab3ad707583f" Oct 30 11:20:11.964977 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.964888 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e483c93f8b697ada1bfd9997c401ecaf63a7859191aaa2124db6ab3ad707583f"} err="failed to get container status \"e483c93f8b697ada1bfd9997c401ecaf63a7859191aaa2124db6ab3ad707583f\": rpc error: code = NotFound desc = could not find container \"e483c93f8b697ada1bfd9997c401ecaf63a7859191aaa2124db6ab3ad707583f\": container with ID starting with e483c93f8b697ada1bfd9997c401ecaf63a7859191aaa2124db6ab3ad707583f not found: ID does not exist" Oct 30 11:20:11.964977 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.964906 4289 scope.go:122] "RemoveContainer" containerID="23bf9a9d69b72e89c854df1e58b2566695814d2d7d2016dcfd609a084cfe3ead" Oct 30 11:20:11.965446 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.965369 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6"] Oct 30 11:20:11.975736 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.975534 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-pzmn6"] Oct 30 11:20:11.975736 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.975563 4289 scope.go:122] "RemoveContainer" containerID="23bf9a9d69b72e89c854df1e58b2566695814d2d7d2016dcfd609a084cfe3ead" Oct 30 11:20:11.975967 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:20:11.975945 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23bf9a9d69b72e89c854df1e58b2566695814d2d7d2016dcfd609a084cfe3ead\": container with ID starting with 23bf9a9d69b72e89c854df1e58b2566695814d2d7d2016dcfd609a084cfe3ead not found: ID does not exist" containerID="23bf9a9d69b72e89c854df1e58b2566695814d2d7d2016dcfd609a084cfe3ead" Oct 30 11:20:11.976084 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:11.976061 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23bf9a9d69b72e89c854df1e58b2566695814d2d7d2016dcfd609a084cfe3ead"} err="failed to get container status \"23bf9a9d69b72e89c854df1e58b2566695814d2d7d2016dcfd609a084cfe3ead\": rpc error: code = NotFound desc = could not find container \"23bf9a9d69b72e89c854df1e58b2566695814d2d7d2016dcfd609a084cfe3ead\": container with ID starting with 23bf9a9d69b72e89c854df1e58b2566695814d2d7d2016dcfd609a084cfe3ead not found: ID does not exist" Oct 30 11:20:12.007763 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.007727 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:12.017526 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.017498 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw"] Oct 30 11:20:12.022102 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.022075 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bdwbw"] Oct 30 11:20:12.062509 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.062484 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74"] Oct 30 11:20:12.066138 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.066114 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-96d74"] Oct 30 11:20:12.074777 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.074757 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:20:12.208354 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.208261 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:12.209145 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.209125 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " Oct 30 11:20:12.209255 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.209175 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-thanos-prometheus-http-client-file\") pod \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " Oct 30 11:20:12.209255 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.209219 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-config\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-config\") pod \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " Oct 30 11:20:12.209255 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.209248 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff8a282-f987-4c65-911d-b56e9ea5dd09-tls-assets\" (UniqueName: \"kubernetes.io/projected/3ff8a282-f987-4c65-911d-b56e9ea5dd09-tls-assets\") pod \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " Oct 30 11:20:12.209414 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.209351 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " Oct 30 11:20:12.209414 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.209388 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff8a282-f987-4c65-911d-b56e9ea5dd09-config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ff8a282-f987-4c65-911d-b56e9ea5dd09-config-out\") pod \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " Oct 30 11:20:12.209535 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.209422 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff8a282-f987-4c65-911d-b56e9ea5dd09-kube-api-access-85c49\" (UniqueName: \"kubernetes.io/projected/3ff8a282-f987-4c65-911d-b56e9ea5dd09-kube-api-access-85c49\") pod \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " Oct 30 11:20:12.209535 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.209490 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-web-config\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-web-config\") pod \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " Oct 30 11:20:12.209757 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.209573 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ff8a282-f987-4c65-911d-b56e9ea5dd09-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3ff8a282-f987-4c65-911d-b56e9ea5dd09-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " Oct 30 11:20:12.209757 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.209606 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-secret-prometheus-web-tls-ca\") pod \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " Oct 30 11:20:12.209757 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.209656 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " Oct 30 11:20:12.209757 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.209689 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ff8a282-f987-4c65-911d-b56e9ea5dd09-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3ff8a282-f987-4c65-911d-b56e9ea5dd09-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " Oct 30 11:20:12.209757 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.209726 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ff8a282-f987-4c65-911d-b56e9ea5dd09-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ff8a282-f987-4c65-911d-b56e9ea5dd09-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\" (UID: \"3ff8a282-f987-4c65-911d-b56e9ea5dd09\") " Oct 30 11:20:12.210720 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.210452 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff8a282-f987-4c65-911d-b56e9ea5dd09-prometheus-data-science-monitoringstack-rulefiles-1" pod "3ff8a282-f987-4c65-911d-b56e9ea5dd09" (UID: "3ff8a282-f987-4c65-911d-b56e9ea5dd09"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:20:12.210720 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.210634 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff8a282-f987-4c65-911d-b56e9ea5dd09-prometheus-data-science-monitoringstack-rulefiles-2" pod "3ff8a282-f987-4c65-911d-b56e9ea5dd09" (UID: "3ff8a282-f987-4c65-911d-b56e9ea5dd09"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:20:12.210875 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.210734 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff8a282-f987-4c65-911d-b56e9ea5dd09-prometheus-data-science-monitoringstack-rulefiles-0" pod "3ff8a282-f987-4c65-911d-b56e9ea5dd09" (UID: "3ff8a282-f987-4c65-911d-b56e9ea5dd09"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:20:12.211915 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.211889 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "3ff8a282-f987-4c65-911d-b56e9ea5dd09" (UID: "3ff8a282-f987-4c65-911d-b56e9ea5dd09"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:20:12.212021 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.211912 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-config" pod "3ff8a282-f987-4c65-911d-b56e9ea5dd09" (UID: "3ff8a282-f987-4c65-911d-b56e9ea5dd09"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:20:12.212315 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.212281 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-thanos-prometheus-http-client-file" pod "3ff8a282-f987-4c65-911d-b56e9ea5dd09" (UID: "3ff8a282-f987-4c65-911d-b56e9ea5dd09"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:20:12.212743 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.212716 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff8a282-f987-4c65-911d-b56e9ea5dd09-tls-assets" pod "3ff8a282-f987-4c65-911d-b56e9ea5dd09" (UID: "3ff8a282-f987-4c65-911d-b56e9ea5dd09"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:20:12.212850 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.212826 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff8a282-f987-4c65-911d-b56e9ea5dd09-kube-api-access-85c49" pod "3ff8a282-f987-4c65-911d-b56e9ea5dd09" (UID: "3ff8a282-f987-4c65-911d-b56e9ea5dd09"). InnerVolumeSpecName "kube-api-access-85c49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:20:12.212989 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.212968 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-secret-prometheus-web-tls-ca" pod "3ff8a282-f987-4c65-911d-b56e9ea5dd09" (UID: "3ff8a282-f987-4c65-911d-b56e9ea5dd09"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:20:12.213307 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.213280 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "3ff8a282-f987-4c65-911d-b56e9ea5dd09" (UID: "3ff8a282-f987-4c65-911d-b56e9ea5dd09"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:20:12.213448 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.213367 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff8a282-f987-4c65-911d-b56e9ea5dd09-config-out" pod "3ff8a282-f987-4c65-911d-b56e9ea5dd09" (UID: "3ff8a282-f987-4c65-911d-b56e9ea5dd09"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:20:12.215563 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.215542 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48" pod "3ff8a282-f987-4c65-911d-b56e9ea5dd09" (UID: "3ff8a282-f987-4c65-911d-b56e9ea5dd09"). InnerVolumeSpecName "pvc-c799cc48-125a-421c-8fc6-8a4d017a7653". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 11:20:12.226288 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.226257 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-web-config" pod "3ff8a282-f987-4c65-911d-b56e9ea5dd09" (UID: "3ff8a282-f987-4c65-911d-b56e9ea5dd09"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:20:12.316398 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.311780 4289 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ff8a282-f987-4c65-911d-b56e9ea5dd09-tls-assets\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:12.316398 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.311838 4289 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") on node \"ip-10-0-137-242.ec2.internal\" " Oct 30 11:20:12.316398 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.311866 4289 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ff8a282-f987-4c65-911d-b56e9ea5dd09-config-out\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:12.316398 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.311881 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85c49\" (UniqueName: \"kubernetes.io/projected/3ff8a282-f987-4c65-911d-b56e9ea5dd09-kube-api-access-85c49\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:12.316398 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.311897 4289 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-web-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:12.316398 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.311913 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3ff8a282-f987-4c65-911d-b56e9ea5dd09-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:12.316398 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.311933 4289 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-secret-prometheus-web-tls-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:12.316398 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.311947 4289 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:12.316398 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.311962 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3ff8a282-f987-4c65-911d-b56e9ea5dd09-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:12.316398 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.311983 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ff8a282-f987-4c65-911d-b56e9ea5dd09-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:12.316398 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.311999 4289 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:12.316398 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.312018 4289 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:12.316398 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.312031 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ff8a282-f987-4c65-911d-b56e9ea5dd09-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:12.326081 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.326059 4289 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-c799cc48-125a-421c-8fc6-8a4d017a7653" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48") on node "ip-10-0-137-242.ec2.internal" Oct 30 11:20:12.412836 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.412806 4289 reconciler_common.go:299] "Volume detached for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"csi-ea06c66241a0a003a5daff5d9fc602f40e53768b00b104c5e4118b92d558c090\"" Oct 30 11:20:12.458818 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.458756 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:20:12.513609 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.513574 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-alertmanager-data-science-monitoringstack-db\") pod \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " Oct 30 11:20:12.513770 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.513640 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-cluster-tls-config\") pod \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " Oct 30 11:20:12.513770 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.513679 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-tls-assets\" (UniqueName: \"kubernetes.io/projected/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-tls-assets\") pod \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " Oct 30 11:20:12.513770 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.513730 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-web-config\" (UniqueName: \"kubernetes.io/secret/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-web-config\") pod \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " Oct 30 11:20:12.513947 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.513778 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-config-out\") pod \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " Oct 30 11:20:12.513947 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.513820 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-kube-api-access-2p8cb\" (UniqueName: \"kubernetes.io/projected/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-kube-api-access-2p8cb\") pod \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " Oct 30 11:20:12.513947 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.513846 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-config-volume\" (UniqueName: \"kubernetes.io/secret/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-config-volume\") pod \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\" (UID: \"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d\") " Oct 30 11:20:12.514091 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.513939 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-alertmanager-data-science-monitoringstack-db" pod "39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d" (UID: "39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:20:12.514091 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.514081 4289 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:12.515950 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.515920 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-tls-assets" pod "39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d" (UID: "39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:20:12.516069 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.515987 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-kube-api-access-2p8cb" pod "39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d" (UID: "39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d"). InnerVolumeSpecName "kube-api-access-2p8cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:20:12.516251 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.516222 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-config-out" pod "39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d" (UID: "39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:20:12.516369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.516314 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-config-volume" pod "39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d" (UID: "39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:20:12.521014 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.520984 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-cluster-tls-config" pod "39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d" (UID: "39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:20:12.528722 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.528697 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-web-config" pod "39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d" (UID: "39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:20:12.615281 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.615244 4289 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-config-out\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:12.615281 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.615275 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p8cb\" (UniqueName: \"kubernetes.io/projected/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-kube-api-access-2p8cb\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:12.615281 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.615289 4289 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:12.615550 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.615302 4289 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-cluster-tls-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:12.615550 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.615314 4289 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-tls-assets\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:12.615550 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.615324 4289 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d-web-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:12.839062 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.839027 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:20:12.839236 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.839215 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="7d1fe067-dea8-40d6-86ca-e19280132f32" containerName="otc-container" containerID="cri-o://516f37ee456e26c8caeadbfcd66ddef5a637ebe5964bff416c4061201b33a9fc" gracePeriod=30 Oct 30 11:20:12.877065 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.877029 4289 generic.go:356] "Generic (PLEG): container finished" podID="3ff8a282-f987-4c65-911d-b56e9ea5dd09" containerID="3c74ee91d769889d58aba84262f48d31c80ce163392f6721505a6fa6d6a2801f" exitCode=0 Oct 30 11:20:12.877233 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.877063 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"3ff8a282-f987-4c65-911d-b56e9ea5dd09","Type":"ContainerDied","Data":"3c74ee91d769889d58aba84262f48d31c80ce163392f6721505a6fa6d6a2801f"} Oct 30 11:20:12.877233 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.877136 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:20:12.877233 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.877154 4289 scope.go:122] "RemoveContainer" containerID="deb60fcd295d51457107e09e69000da518ab9300141e985a5a0b26ebae061751" Oct 30 11:20:12.877490 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.877140 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"3ff8a282-f987-4c65-911d-b56e9ea5dd09","Type":"ContainerDied","Data":"b2b07c0bcb37e5b868f778136c4ec08eee231084e2147fa677930b4adac939ab"} Oct 30 11:20:12.879600 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.879574 4289 generic.go:356] "Generic (PLEG): container finished" podID="39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d" containerID="446d7d8d01226fc914207861ccaf24083c827da98d224fbe74ca6bcb4689c130" exitCode=0 Oct 30 11:20:12.879722 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.879612 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d","Type":"ContainerDied","Data":"446d7d8d01226fc914207861ccaf24083c827da98d224fbe74ca6bcb4689c130"} Oct 30 11:20:12.879722 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.879660 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d","Type":"ContainerDied","Data":"8ba2a386fa63e9240f9117dca49803a7c55de6e8ca90012dafd9ce8032031dfc"} Oct 30 11:20:12.879857 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.879753 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:20:12.936041 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.935996 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:12.952925 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.952899 4289 scope.go:122] "RemoveContainer" containerID="3c74ee91d769889d58aba84262f48d31c80ce163392f6721505a6fa6d6a2801f" Oct 30 11:20:12.962155 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.962125 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 11:20:12.963702 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.963679 4289 scope.go:122] "RemoveContainer" containerID="1420776adc26b76c076fcb26b0435d0e8892bcfb9a12dc85939a3b6ff762a564" Oct 30 11:20:12.970701 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.970678 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 11:20:12.974398 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.974377 4289 scope.go:122] "RemoveContainer" containerID="6c8c73f3012976ca88fe74582857a7159373e77c2e1f8ea481504d3b36a533cf" Oct 30 11:20:12.982196 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.982179 4289 scope.go:122] "RemoveContainer" containerID="deb60fcd295d51457107e09e69000da518ab9300141e985a5a0b26ebae061751" Oct 30 11:20:12.982502 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:20:12.982476 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deb60fcd295d51457107e09e69000da518ab9300141e985a5a0b26ebae061751\": container with ID starting with deb60fcd295d51457107e09e69000da518ab9300141e985a5a0b26ebae061751 not found: ID does not exist" containerID="deb60fcd295d51457107e09e69000da518ab9300141e985a5a0b26ebae061751" Oct 30 11:20:12.982597 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.982516 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deb60fcd295d51457107e09e69000da518ab9300141e985a5a0b26ebae061751"} err="failed to get container status \"deb60fcd295d51457107e09e69000da518ab9300141e985a5a0b26ebae061751\": rpc error: code = NotFound desc = could not find container \"deb60fcd295d51457107e09e69000da518ab9300141e985a5a0b26ebae061751\": container with ID starting with deb60fcd295d51457107e09e69000da518ab9300141e985a5a0b26ebae061751 not found: ID does not exist" Oct 30 11:20:12.982597 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.982543 4289 scope.go:122] "RemoveContainer" containerID="3c74ee91d769889d58aba84262f48d31c80ce163392f6721505a6fa6d6a2801f" Oct 30 11:20:12.982827 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:20:12.982803 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c74ee91d769889d58aba84262f48d31c80ce163392f6721505a6fa6d6a2801f\": container with ID starting with 3c74ee91d769889d58aba84262f48d31c80ce163392f6721505a6fa6d6a2801f not found: ID does not exist" containerID="3c74ee91d769889d58aba84262f48d31c80ce163392f6721505a6fa6d6a2801f" Oct 30 11:20:12.982944 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.982833 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c74ee91d769889d58aba84262f48d31c80ce163392f6721505a6fa6d6a2801f"} err="failed to get container status \"3c74ee91d769889d58aba84262f48d31c80ce163392f6721505a6fa6d6a2801f\": rpc error: code = NotFound desc = could not find container \"3c74ee91d769889d58aba84262f48d31c80ce163392f6721505a6fa6d6a2801f\": container with ID starting with 3c74ee91d769889d58aba84262f48d31c80ce163392f6721505a6fa6d6a2801f not found: ID does not exist" Oct 30 11:20:12.982944 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.982851 4289 scope.go:122] "RemoveContainer" containerID="1420776adc26b76c076fcb26b0435d0e8892bcfb9a12dc85939a3b6ff762a564" Oct 30 11:20:12.983074 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:20:12.983058 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1420776adc26b76c076fcb26b0435d0e8892bcfb9a12dc85939a3b6ff762a564\": container with ID starting with 1420776adc26b76c076fcb26b0435d0e8892bcfb9a12dc85939a3b6ff762a564 not found: ID does not exist" containerID="1420776adc26b76c076fcb26b0435d0e8892bcfb9a12dc85939a3b6ff762a564" Oct 30 11:20:12.983132 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.983075 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1420776adc26b76c076fcb26b0435d0e8892bcfb9a12dc85939a3b6ff762a564"} err="failed to get container status \"1420776adc26b76c076fcb26b0435d0e8892bcfb9a12dc85939a3b6ff762a564\": rpc error: code = NotFound desc = could not find container \"1420776adc26b76c076fcb26b0435d0e8892bcfb9a12dc85939a3b6ff762a564\": container with ID starting with 1420776adc26b76c076fcb26b0435d0e8892bcfb9a12dc85939a3b6ff762a564 not found: ID does not exist" Oct 30 11:20:12.983132 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.983096 4289 scope.go:122] "RemoveContainer" containerID="6c8c73f3012976ca88fe74582857a7159373e77c2e1f8ea481504d3b36a533cf" Oct 30 11:20:12.983359 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:20:12.983338 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c8c73f3012976ca88fe74582857a7159373e77c2e1f8ea481504d3b36a533cf\": container with ID starting with 6c8c73f3012976ca88fe74582857a7159373e77c2e1f8ea481504d3b36a533cf not found: ID does not exist" containerID="6c8c73f3012976ca88fe74582857a7159373e77c2e1f8ea481504d3b36a533cf" Oct 30 11:20:12.983414 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.983364 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c8c73f3012976ca88fe74582857a7159373e77c2e1f8ea481504d3b36a533cf"} err="failed to get container status \"6c8c73f3012976ca88fe74582857a7159373e77c2e1f8ea481504d3b36a533cf\": rpc error: code = NotFound desc = could not find container \"6c8c73f3012976ca88fe74582857a7159373e77c2e1f8ea481504d3b36a533cf\": container with ID starting with 6c8c73f3012976ca88fe74582857a7159373e77c2e1f8ea481504d3b36a533cf not found: ID does not exist" Oct 30 11:20:12.983414 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.983381 4289 scope.go:122] "RemoveContainer" containerID="288302b0523fb8b3fc3761bf69b697aeda8bfb9c9b17c819cf5bc439fc2a6fce" Oct 30 11:20:12.992820 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:12.992803 4289 scope.go:122] "RemoveContainer" containerID="446d7d8d01226fc914207861ccaf24083c827da98d224fbe74ca6bcb4689c130" Oct 30 11:20:13.000853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.000834 4289 scope.go:122] "RemoveContainer" containerID="e9dab2b1e95a7eaa39c06e4cb981e9435a37b92cd85caeebf1a43a377d5c0681" Oct 30 11:20:13.008803 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.008780 4289 scope.go:122] "RemoveContainer" containerID="288302b0523fb8b3fc3761bf69b697aeda8bfb9c9b17c819cf5bc439fc2a6fce" Oct 30 11:20:13.009078 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:20:13.009057 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"288302b0523fb8b3fc3761bf69b697aeda8bfb9c9b17c819cf5bc439fc2a6fce\": container with ID starting with 288302b0523fb8b3fc3761bf69b697aeda8bfb9c9b17c819cf5bc439fc2a6fce not found: ID does not exist" containerID="288302b0523fb8b3fc3761bf69b697aeda8bfb9c9b17c819cf5bc439fc2a6fce" Oct 30 11:20:13.009151 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.009084 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"288302b0523fb8b3fc3761bf69b697aeda8bfb9c9b17c819cf5bc439fc2a6fce"} err="failed to get container status \"288302b0523fb8b3fc3761bf69b697aeda8bfb9c9b17c819cf5bc439fc2a6fce\": rpc error: code = NotFound desc = could not find container \"288302b0523fb8b3fc3761bf69b697aeda8bfb9c9b17c819cf5bc439fc2a6fce\": container with ID starting with 288302b0523fb8b3fc3761bf69b697aeda8bfb9c9b17c819cf5bc439fc2a6fce not found: ID does not exist" Oct 30 11:20:13.009151 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.009107 4289 scope.go:122] "RemoveContainer" containerID="446d7d8d01226fc914207861ccaf24083c827da98d224fbe74ca6bcb4689c130" Oct 30 11:20:13.009400 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:20:13.009378 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"446d7d8d01226fc914207861ccaf24083c827da98d224fbe74ca6bcb4689c130\": container with ID starting with 446d7d8d01226fc914207861ccaf24083c827da98d224fbe74ca6bcb4689c130 not found: ID does not exist" containerID="446d7d8d01226fc914207861ccaf24083c827da98d224fbe74ca6bcb4689c130" Oct 30 11:20:13.009488 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.009405 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"446d7d8d01226fc914207861ccaf24083c827da98d224fbe74ca6bcb4689c130"} err="failed to get container status \"446d7d8d01226fc914207861ccaf24083c827da98d224fbe74ca6bcb4689c130\": rpc error: code = NotFound desc = could not find container \"446d7d8d01226fc914207861ccaf24083c827da98d224fbe74ca6bcb4689c130\": container with ID starting with 446d7d8d01226fc914207861ccaf24083c827da98d224fbe74ca6bcb4689c130 not found: ID does not exist" Oct 30 11:20:13.009488 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.009420 4289 scope.go:122] "RemoveContainer" containerID="e9dab2b1e95a7eaa39c06e4cb981e9435a37b92cd85caeebf1a43a377d5c0681" Oct 30 11:20:13.009792 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:20:13.009769 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9dab2b1e95a7eaa39c06e4cb981e9435a37b92cd85caeebf1a43a377d5c0681\": container with ID starting with e9dab2b1e95a7eaa39c06e4cb981e9435a37b92cd85caeebf1a43a377d5c0681 not found: ID does not exist" containerID="e9dab2b1e95a7eaa39c06e4cb981e9435a37b92cd85caeebf1a43a377d5c0681" Oct 30 11:20:13.009851 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.009799 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9dab2b1e95a7eaa39c06e4cb981e9435a37b92cd85caeebf1a43a377d5c0681"} err="failed to get container status \"e9dab2b1e95a7eaa39c06e4cb981e9435a37b92cd85caeebf1a43a377d5c0681\": rpc error: code = NotFound desc = could not find container \"e9dab2b1e95a7eaa39c06e4cb981e9435a37b92cd85caeebf1a43a377d5c0681\": container with ID starting with e9dab2b1e95a7eaa39c06e4cb981e9435a37b92cd85caeebf1a43a377d5c0681 not found: ID does not exist" Oct 30 11:20:13.015646 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.015622 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:20:13.020115 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.020090 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:20:13.095812 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.095753 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:20:13.117104 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.117058 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:13.219702 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.219675 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d1fe067-dea8-40d6-86ca-e19280132f32-kube-api-access-mfrgk\" (UniqueName: \"kubernetes.io/projected/7d1fe067-dea8-40d6-86ca-e19280132f32-kube-api-access-mfrgk\") pod \"7d1fe067-dea8-40d6-86ca-e19280132f32\" (UID: \"7d1fe067-dea8-40d6-86ca-e19280132f32\") " Oct 30 11:20:13.219870 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.219730 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d1fe067-dea8-40d6-86ca-e19280132f32-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1fe067-dea8-40d6-86ca-e19280132f32-monitoring-tls-certs\") pod \"7d1fe067-dea8-40d6-86ca-e19280132f32\" (UID: \"7d1fe067-dea8-40d6-86ca-e19280132f32\") " Oct 30 11:20:13.219870 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.219754 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d1fe067-dea8-40d6-86ca-e19280132f32-otc-internal\" (UniqueName: \"kubernetes.io/configmap/7d1fe067-dea8-40d6-86ca-e19280132f32-otc-internal\") pod \"7d1fe067-dea8-40d6-86ca-e19280132f32\" (UID: \"7d1fe067-dea8-40d6-86ca-e19280132f32\") " Oct 30 11:20:13.219870 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.219794 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d1fe067-dea8-40d6-86ca-e19280132f32-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7d1fe067-dea8-40d6-86ca-e19280132f32-data-science-collector-ta-client-cert\") pod \"7d1fe067-dea8-40d6-86ca-e19280132f32\" (UID: \"7d1fe067-dea8-40d6-86ca-e19280132f32\") " Oct 30 11:20:13.219870 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.219819 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d1fe067-dea8-40d6-86ca-e19280132f32-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1fe067-dea8-40d6-86ca-e19280132f32-tls-certs\") pod \"7d1fe067-dea8-40d6-86ca-e19280132f32\" (UID: \"7d1fe067-dea8-40d6-86ca-e19280132f32\") " Oct 30 11:20:13.220218 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.220188 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d1fe067-dea8-40d6-86ca-e19280132f32-otc-internal" pod "7d1fe067-dea8-40d6-86ca-e19280132f32" (UID: "7d1fe067-dea8-40d6-86ca-e19280132f32"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:20:13.221962 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.221932 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d1fe067-dea8-40d6-86ca-e19280132f32-data-science-collector-ta-client-cert" pod "7d1fe067-dea8-40d6-86ca-e19280132f32" (UID: "7d1fe067-dea8-40d6-86ca-e19280132f32"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:20:13.221962 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.221956 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1fe067-dea8-40d6-86ca-e19280132f32-kube-api-access-mfrgk" pod "7d1fe067-dea8-40d6-86ca-e19280132f32" (UID: "7d1fe067-dea8-40d6-86ca-e19280132f32"). InnerVolumeSpecName "kube-api-access-mfrgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:20:13.222099 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.221962 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d1fe067-dea8-40d6-86ca-e19280132f32-tls-certs" pod "7d1fe067-dea8-40d6-86ca-e19280132f32" (UID: "7d1fe067-dea8-40d6-86ca-e19280132f32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:20:13.222099 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.221945 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d1fe067-dea8-40d6-86ca-e19280132f32-monitoring-tls-certs" pod "7d1fe067-dea8-40d6-86ca-e19280132f32" (UID: "7d1fe067-dea8-40d6-86ca-e19280132f32"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:20:13.321322 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.321292 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1fe067-dea8-40d6-86ca-e19280132f32-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:13.321322 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.321319 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7d1fe067-dea8-40d6-86ca-e19280132f32-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:13.321322 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.321330 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7d1fe067-dea8-40d6-86ca-e19280132f32-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:13.321593 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.321341 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1fe067-dea8-40d6-86ca-e19280132f32-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:13.321593 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.321351 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfrgk\" (UniqueName: \"kubernetes.io/projected/7d1fe067-dea8-40d6-86ca-e19280132f32-kube-api-access-mfrgk\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:20:13.625633 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.625603 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d" path="/var/lib/kubelet/pods/39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d/volumes" Oct 30 11:20:13.626052 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.626039 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ba78eac-b3ef-4dc0-adf4-37897ca5b735" path="/var/lib/kubelet/pods/3ba78eac-b3ef-4dc0-adf4-37897ca5b735/volumes" Oct 30 11:20:13.626451 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.626424 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ff8a282-f987-4c65-911d-b56e9ea5dd09" path="/var/lib/kubelet/pods/3ff8a282-f987-4c65-911d-b56e9ea5dd09/volumes" Oct 30 11:20:13.627174 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.627160 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51c34556-aa72-4eed-ac1c-eb3f4534a62a" path="/var/lib/kubelet/pods/51c34556-aa72-4eed-ac1c-eb3f4534a62a/volumes" Oct 30 11:20:13.627585 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.627573 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f4aed18-f28d-482c-a868-229cdadff6f3" path="/var/lib/kubelet/pods/8f4aed18-f28d-482c-a868-229cdadff6f3/volumes" Oct 30 11:20:13.628252 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.628238 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e473baa6-a55e-4bce-acb3-267ef3c8ad7e" path="/var/lib/kubelet/pods/e473baa6-a55e-4bce-acb3-267ef3c8ad7e/volumes" Oct 30 11:20:13.797759 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.797709 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:13.888132 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.888053 4289 generic.go:356] "Generic (PLEG): container finished" podID="7d1fe067-dea8-40d6-86ca-e19280132f32" containerID="516f37ee456e26c8caeadbfcd66ddef5a637ebe5964bff416c4061201b33a9fc" exitCode=0 Oct 30 11:20:13.888132 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.888100 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:20:13.888308 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.888127 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7d1fe067-dea8-40d6-86ca-e19280132f32","Type":"ContainerDied","Data":"516f37ee456e26c8caeadbfcd66ddef5a637ebe5964bff416c4061201b33a9fc"} Oct 30 11:20:13.888308 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.888175 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7d1fe067-dea8-40d6-86ca-e19280132f32","Type":"ContainerDied","Data":"d8ee853870830d38ac78ba9dd5318339d0f4a80f27977b5069c991d126cfe7ea"} Oct 30 11:20:13.888308 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.888196 4289 scope.go:122] "RemoveContainer" containerID="516f37ee456e26c8caeadbfcd66ddef5a637ebe5964bff416c4061201b33a9fc" Oct 30 11:20:13.896440 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.896408 4289 scope.go:122] "RemoveContainer" containerID="516f37ee456e26c8caeadbfcd66ddef5a637ebe5964bff416c4061201b33a9fc" Oct 30 11:20:13.896804 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:20:13.896781 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"516f37ee456e26c8caeadbfcd66ddef5a637ebe5964bff416c4061201b33a9fc\": container with ID starting with 516f37ee456e26c8caeadbfcd66ddef5a637ebe5964bff416c4061201b33a9fc not found: ID does not exist" containerID="516f37ee456e26c8caeadbfcd66ddef5a637ebe5964bff416c4061201b33a9fc" Oct 30 11:20:13.896891 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.896812 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"516f37ee456e26c8caeadbfcd66ddef5a637ebe5964bff416c4061201b33a9fc"} err="failed to get container status \"516f37ee456e26c8caeadbfcd66ddef5a637ebe5964bff416c4061201b33a9fc\": rpc error: code = NotFound desc = could not find container \"516f37ee456e26c8caeadbfcd66ddef5a637ebe5964bff416c4061201b33a9fc\": container with ID starting with 516f37ee456e26c8caeadbfcd66ddef5a637ebe5964bff416c4061201b33a9fc not found: ID does not exist" Oct 30 11:20:13.940847 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.940820 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:20:13.943018 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.942997 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:20:13.965802 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:13.965765 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:14.130590 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:14.130548 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:15.624324 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:15.624291 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d1fe067-dea8-40d6-86ca-e19280132f32" path="/var/lib/kubelet/pods/7d1fe067-dea8-40d6-86ca-e19280132f32/volumes" Oct 30 11:20:18.034725 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.034690 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:18.207239 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.207202 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:18.275899 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.275870 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts"] Oct 30 11:20:18.276199 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276185 4289 cpu_manager.go:414] "containerMap: removing container" podUID="51c34556-aa72-4eed-ac1c-eb3f4534a62a" containerName="kube-rbac-proxy" Oct 30 11:20:18.276252 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276201 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51c34556-aa72-4eed-ac1c-eb3f4534a62a" containerName="kube-rbac-proxy" Oct 30 11:20:18.276252 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276209 4289 cpu_manager.go:414] "containerMap: removing container" podUID="51c34556-aa72-4eed-ac1c-eb3f4534a62a" containerName="method-gate" Oct 30 11:20:18.276252 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276215 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51c34556-aa72-4eed-ac1c-eb3f4534a62a" containerName="method-gate" Oct 30 11:20:18.276252 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276224 4289 cpu_manager.go:414] "containerMap: removing container" podUID="e473baa6-a55e-4bce-acb3-267ef3c8ad7e" containerName="korrel8r" Oct 30 11:20:18.276252 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276229 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e473baa6-a55e-4bce-acb3-267ef3c8ad7e" containerName="korrel8r" Oct 30 11:20:18.276252 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276238 4289 cpu_manager.go:414] "containerMap: removing container" podUID="39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d" containerName="init-config-reloader" Oct 30 11:20:18.276252 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276242 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d" containerName="init-config-reloader" Oct 30 11:20:18.276252 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276251 4289 cpu_manager.go:414] "containerMap: removing container" podUID="3ff8a282-f987-4c65-911d-b56e9ea5dd09" containerName="config-reloader" Oct 30 11:20:18.276252 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276255 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff8a282-f987-4c65-911d-b56e9ea5dd09" containerName="config-reloader" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276263 4289 cpu_manager.go:414] "containerMap: removing container" podUID="3ba78eac-b3ef-4dc0-adf4-37897ca5b735" containerName="kube-rbac-proxy" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276268 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ba78eac-b3ef-4dc0-adf4-37897ca5b735" containerName="kube-rbac-proxy" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276274 4289 cpu_manager.go:414] "containerMap: removing container" podUID="39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d" containerName="config-reloader" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276279 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d" containerName="config-reloader" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276286 4289 cpu_manager.go:414] "containerMap: removing container" podUID="3ff8a282-f987-4c65-911d-b56e9ea5dd09" containerName="init-config-reloader" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276290 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff8a282-f987-4c65-911d-b56e9ea5dd09" containerName="init-config-reloader" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276300 4289 cpu_manager.go:414] "containerMap: removing container" podUID="8f4aed18-f28d-482c-a868-229cdadff6f3" containerName="prom-label-proxy" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276304 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f4aed18-f28d-482c-a868-229cdadff6f3" containerName="prom-label-proxy" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276310 4289 cpu_manager.go:414] "containerMap: removing container" podUID="3ff8a282-f987-4c65-911d-b56e9ea5dd09" containerName="thanos-sidecar" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276314 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff8a282-f987-4c65-911d-b56e9ea5dd09" containerName="thanos-sidecar" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276323 4289 cpu_manager.go:414] "containerMap: removing container" podUID="7d1fe067-dea8-40d6-86ca-e19280132f32" containerName="otc-container" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276327 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d1fe067-dea8-40d6-86ca-e19280132f32" containerName="otc-container" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276337 4289 cpu_manager.go:414] "containerMap: removing container" podUID="51c34556-aa72-4eed-ac1c-eb3f4534a62a" containerName="prom-label-proxy" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276342 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51c34556-aa72-4eed-ac1c-eb3f4534a62a" containerName="prom-label-proxy" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276347 4289 cpu_manager.go:414] "containerMap: removing container" podUID="3ff8a282-f987-4c65-911d-b56e9ea5dd09" containerName="prometheus" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276351 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff8a282-f987-4c65-911d-b56e9ea5dd09" containerName="prometheus" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276359 4289 cpu_manager.go:414] "containerMap: removing container" podUID="8f4aed18-f28d-482c-a868-229cdadff6f3" containerName="kube-rbac-proxy" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276364 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f4aed18-f28d-482c-a868-229cdadff6f3" containerName="kube-rbac-proxy" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276369 4289 cpu_manager.go:414] "containerMap: removing container" podUID="39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d" containerName="alertmanager" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276373 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d" containerName="alertmanager" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276380 4289 cpu_manager.go:414] "containerMap: removing container" podUID="8f4aed18-f28d-482c-a868-229cdadff6f3" containerName="method-gate" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276384 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f4aed18-f28d-482c-a868-229cdadff6f3" containerName="method-gate" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276444 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="51c34556-aa72-4eed-ac1c-eb3f4534a62a" containerName="prom-label-proxy" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276454 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff8a282-f987-4c65-911d-b56e9ea5dd09" containerName="prometheus" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276464 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f4aed18-f28d-482c-a868-229cdadff6f3" containerName="kube-rbac-proxy" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276472 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d" containerName="alertmanager" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276482 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f4aed18-f28d-482c-a868-229cdadff6f3" containerName="method-gate" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276491 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="51c34556-aa72-4eed-ac1c-eb3f4534a62a" containerName="kube-rbac-proxy" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276499 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="51c34556-aa72-4eed-ac1c-eb3f4534a62a" containerName="method-gate" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276505 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="e473baa6-a55e-4bce-acb3-267ef3c8ad7e" containerName="korrel8r" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276511 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff8a282-f987-4c65-911d-b56e9ea5dd09" containerName="config-reloader" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276516 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ba78eac-b3ef-4dc0-adf4-37897ca5b735" containerName="kube-rbac-proxy" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276521 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="39bbe9d3-25a1-4d9b-8eca-dc33e1fd455d" containerName="config-reloader" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276526 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f4aed18-f28d-482c-a868-229cdadff6f3" containerName="prom-label-proxy" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276531 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff8a282-f987-4c65-911d-b56e9ea5dd09" containerName="thanos-sidecar" Oct 30 11:20:18.276531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.276536 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d1fe067-dea8-40d6-86ca-e19280132f32" containerName="otc-container" Oct 30 11:20:18.281161 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.281141 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" Oct 30 11:20:18.308871 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.308841 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 11:20:18.309129 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.309110 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 11:20:18.310081 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.310058 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts"] Oct 30 11:20:18.311506 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.311490 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 11:20:18.311570 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.311533 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 11:20:18.311570 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.311541 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 11:20:18.311570 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.311554 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-gbr8d\"" Oct 30 11:20:18.311679 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.311600 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 11:20:18.327052 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.327028 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf"] Oct 30 11:20:18.330387 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.330367 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" Oct 30 11:20:18.352363 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.351213 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-xw8qg\"" Oct 30 11:20:18.352363 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.351067 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 11:20:18.353002 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.352981 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf"] Oct 30 11:20:18.359357 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.359335 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd"] Oct 30 11:20:18.366821 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.366011 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/22a70032-0628-4014-8cf5-aaac0fe64900-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2wfts\" (UID: \"22a70032-0628-4014-8cf5-aaac0fe64900\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" Oct 30 11:20:18.366821 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.366083 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22a70032-0628-4014-8cf5-aaac0fe64900-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2wfts\" (UID: \"22a70032-0628-4014-8cf5-aaac0fe64900\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" Oct 30 11:20:18.366821 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.366118 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/22a70032-0628-4014-8cf5-aaac0fe64900-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2wfts\" (UID: \"22a70032-0628-4014-8cf5-aaac0fe64900\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" Oct 30 11:20:18.366821 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.366153 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/22a70032-0628-4014-8cf5-aaac0fe64900-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2wfts\" (UID: \"22a70032-0628-4014-8cf5-aaac0fe64900\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" Oct 30 11:20:18.366821 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.366217 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mljts\" (UniqueName: \"kubernetes.io/projected/22a70032-0628-4014-8cf5-aaac0fe64900-kube-api-access-mljts\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2wfts\" (UID: \"22a70032-0628-4014-8cf5-aaac0fe64900\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" Oct 30 11:20:18.366821 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.366271 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" Oct 30 11:20:18.384961 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.384933 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd"] Oct 30 11:20:18.395154 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.395123 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:18.400088 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.400059 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 11:20:18.400215 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.400095 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 11:20:18.400215 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.400100 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 11:20:18.400215 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.400176 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-2s6x2\"" Oct 30 11:20:18.400331 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.400224 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 11:20:18.403964 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.403941 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx"] Oct 30 11:20:18.407640 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.407623 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx" Oct 30 11:20:18.425332 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.425306 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx"] Oct 30 11:20:18.466627 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.466596 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/22a70032-0628-4014-8cf5-aaac0fe64900-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2wfts\" (UID: \"22a70032-0628-4014-8cf5-aaac0fe64900\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" Oct 30 11:20:18.466760 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.466634 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a33defc9-4a83-4e81-9095-aa32e383792c-config\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf\" (UID: \"a33defc9-4a83-4e81-9095-aa32e383792c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" Oct 30 11:20:18.466760 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.466662 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t76pk\" (UniqueName: \"kubernetes.io/projected/a33defc9-4a83-4e81-9095-aa32e383792c-kube-api-access-t76pk\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf\" (UID: \"a33defc9-4a83-4e81-9095-aa32e383792c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" Oct 30 11:20:18.466760 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.466739 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e962d8b6-4549-4ec3-8e7e-bf003453f7a0-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-bh5wd\" (UID: \"e962d8b6-4549-4ec3-8e7e-bf003453f7a0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" Oct 30 11:20:18.466876 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.466801 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6prsv\" (UniqueName: \"kubernetes.io/projected/e962d8b6-4549-4ec3-8e7e-bf003453f7a0-kube-api-access-6prsv\") pod \"data-science-thanos-querier-proxy-7b56895dd5-bh5wd\" (UID: \"e962d8b6-4549-4ec3-8e7e-bf003453f7a0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" Oct 30 11:20:18.466876 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.466826 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e962d8b6-4549-4ec3-8e7e-bf003453f7a0-config\") pod \"data-science-thanos-querier-proxy-7b56895dd5-bh5wd\" (UID: \"e962d8b6-4549-4ec3-8e7e-bf003453f7a0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" Oct 30 11:20:18.466876 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.466851 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22a70032-0628-4014-8cf5-aaac0fe64900-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2wfts\" (UID: \"22a70032-0628-4014-8cf5-aaac0fe64900\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" Oct 30 11:20:18.466967 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.466896 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/22a70032-0628-4014-8cf5-aaac0fe64900-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2wfts\" (UID: \"22a70032-0628-4014-8cf5-aaac0fe64900\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" Oct 30 11:20:18.467034 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.467007 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/22a70032-0628-4014-8cf5-aaac0fe64900-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2wfts\" (UID: \"22a70032-0628-4014-8cf5-aaac0fe64900\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" Oct 30 11:20:18.467081 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.467069 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a33defc9-4a83-4e81-9095-aa32e383792c-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf\" (UID: \"a33defc9-4a83-4e81-9095-aa32e383792c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" Oct 30 11:20:18.467114 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:20:18.467079 4289 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 30 11:20:18.467114 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.467093 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/a33defc9-4a83-4e81-9095-aa32e383792c-client-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf\" (UID: \"a33defc9-4a83-4e81-9095-aa32e383792c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" Oct 30 11:20:18.467198 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.467120 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/a33defc9-4a83-4e81-9095-aa32e383792c-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf\" (UID: \"a33defc9-4a83-4e81-9095-aa32e383792c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" Oct 30 11:20:18.467198 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:20:18.467140 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a70032-0628-4014-8cf5-aaac0fe64900-prometheus-client-cert podName:22a70032-0628-4014-8cf5-aaac0fe64900 nodeName:}" failed. No retries permitted until 2026-10-30 11:20:18.967120191 +0000 UTC m=+2109.468499668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/22a70032-0628-4014-8cf5-aaac0fe64900-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-2wfts" (UID: "22a70032-0628-4014-8cf5-aaac0fe64900") : secret "prometheus-operated-tls" not found Oct 30 11:20:18.467305 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.467203 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mljts\" (UniqueName: \"kubernetes.io/projected/22a70032-0628-4014-8cf5-aaac0fe64900-kube-api-access-mljts\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2wfts\" (UID: \"22a70032-0628-4014-8cf5-aaac0fe64900\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" Oct 30 11:20:18.467479 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.467454 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/22a70032-0628-4014-8cf5-aaac0fe64900-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2wfts\" (UID: \"22a70032-0628-4014-8cf5-aaac0fe64900\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" Oct 30 11:20:18.469462 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.469441 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22a70032-0628-4014-8cf5-aaac0fe64900-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2wfts\" (UID: \"22a70032-0628-4014-8cf5-aaac0fe64900\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" Oct 30 11:20:18.469632 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.469616 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/22a70032-0628-4014-8cf5-aaac0fe64900-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2wfts\" (UID: \"22a70032-0628-4014-8cf5-aaac0fe64900\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" Oct 30 11:20:18.489671 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.489647 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mljts\" (UniqueName: \"kubernetes.io/projected/22a70032-0628-4014-8cf5-aaac0fe64900-kube-api-access-mljts\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2wfts\" (UID: \"22a70032-0628-4014-8cf5-aaac0fe64900\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" Oct 30 11:20:18.500194 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.500164 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 11:20:18.500194 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.500181 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 11:20:18.500318 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.500228 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-tj74z\"" Oct 30 11:20:18.568242 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.568169 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e962d8b6-4549-4ec3-8e7e-bf003453f7a0-config\") pod \"data-science-thanos-querier-proxy-7b56895dd5-bh5wd\" (UID: \"e962d8b6-4549-4ec3-8e7e-bf003453f7a0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" Oct 30 11:20:18.568242 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.568212 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1b34e08e-81fb-4d17-8f15-2e24e94f47c0-tls\") pod \"korrel8r-5cbcc89b8f-lcljx\" (UID: \"1b34e08e-81fb-4d17-8f15-2e24e94f47c0\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx" Oct 30 11:20:18.568242 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.568230 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf6n5\" (UniqueName: \"kubernetes.io/projected/1b34e08e-81fb-4d17-8f15-2e24e94f47c0-kube-api-access-kf6n5\") pod \"korrel8r-5cbcc89b8f-lcljx\" (UID: \"1b34e08e-81fb-4d17-8f15-2e24e94f47c0\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx" Oct 30 11:20:18.568423 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.568327 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a33defc9-4a83-4e81-9095-aa32e383792c-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf\" (UID: \"a33defc9-4a83-4e81-9095-aa32e383792c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" Oct 30 11:20:18.568423 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.568360 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/a33defc9-4a83-4e81-9095-aa32e383792c-client-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf\" (UID: \"a33defc9-4a83-4e81-9095-aa32e383792c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" Oct 30 11:20:18.568423 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.568381 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/a33defc9-4a83-4e81-9095-aa32e383792c-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf\" (UID: \"a33defc9-4a83-4e81-9095-aa32e383792c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" Oct 30 11:20:18.568577 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.568418 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b34e08e-81fb-4d17-8f15-2e24e94f47c0-config\") pod \"korrel8r-5cbcc89b8f-lcljx\" (UID: \"1b34e08e-81fb-4d17-8f15-2e24e94f47c0\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx" Oct 30 11:20:18.568577 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.568492 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a33defc9-4a83-4e81-9095-aa32e383792c-config\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf\" (UID: \"a33defc9-4a83-4e81-9095-aa32e383792c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" Oct 30 11:20:18.568577 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.568509 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t76pk\" (UniqueName: \"kubernetes.io/projected/a33defc9-4a83-4e81-9095-aa32e383792c-kube-api-access-t76pk\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf\" (UID: \"a33defc9-4a83-4e81-9095-aa32e383792c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" Oct 30 11:20:18.568577 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.568565 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e962d8b6-4549-4ec3-8e7e-bf003453f7a0-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-bh5wd\" (UID: \"e962d8b6-4549-4ec3-8e7e-bf003453f7a0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" Oct 30 11:20:18.568783 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.568604 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6prsv\" (UniqueName: \"kubernetes.io/projected/e962d8b6-4549-4ec3-8e7e-bf003453f7a0-kube-api-access-6prsv\") pod \"data-science-thanos-querier-proxy-7b56895dd5-bh5wd\" (UID: \"e962d8b6-4549-4ec3-8e7e-bf003453f7a0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" Oct 30 11:20:18.568984 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.568960 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/a33defc9-4a83-4e81-9095-aa32e383792c-client-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf\" (UID: \"a33defc9-4a83-4e81-9095-aa32e383792c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" Oct 30 11:20:18.569099 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.569077 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e962d8b6-4549-4ec3-8e7e-bf003453f7a0-config\") pod \"data-science-thanos-querier-proxy-7b56895dd5-bh5wd\" (UID: \"e962d8b6-4549-4ec3-8e7e-bf003453f7a0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" Oct 30 11:20:18.569162 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.569087 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/a33defc9-4a83-4e81-9095-aa32e383792c-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf\" (UID: \"a33defc9-4a83-4e81-9095-aa32e383792c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" Oct 30 11:20:18.569318 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.569301 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a33defc9-4a83-4e81-9095-aa32e383792c-config\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf\" (UID: \"a33defc9-4a83-4e81-9095-aa32e383792c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" Oct 30 11:20:18.570924 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.570898 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a33defc9-4a83-4e81-9095-aa32e383792c-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf\" (UID: \"a33defc9-4a83-4e81-9095-aa32e383792c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" Oct 30 11:20:18.571067 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.571048 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e962d8b6-4549-4ec3-8e7e-bf003453f7a0-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-bh5wd\" (UID: \"e962d8b6-4549-4ec3-8e7e-bf003453f7a0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" Oct 30 11:20:18.591298 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.591272 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6prsv\" (UniqueName: \"kubernetes.io/projected/e962d8b6-4549-4ec3-8e7e-bf003453f7a0-kube-api-access-6prsv\") pod \"data-science-thanos-querier-proxy-7b56895dd5-bh5wd\" (UID: \"e962d8b6-4549-4ec3-8e7e-bf003453f7a0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" Oct 30 11:20:18.591473 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.591455 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t76pk\" (UniqueName: \"kubernetes.io/projected/a33defc9-4a83-4e81-9095-aa32e383792c-kube-api-access-t76pk\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf\" (UID: \"a33defc9-4a83-4e81-9095-aa32e383792c\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" Oct 30 11:20:18.647815 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.647788 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" Oct 30 11:20:18.667209 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.667176 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:18.669248 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.669225 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b34e08e-81fb-4d17-8f15-2e24e94f47c0-config\") pod \"korrel8r-5cbcc89b8f-lcljx\" (UID: \"1b34e08e-81fb-4d17-8f15-2e24e94f47c0\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx" Oct 30 11:20:18.669360 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.669310 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1b34e08e-81fb-4d17-8f15-2e24e94f47c0-tls\") pod \"korrel8r-5cbcc89b8f-lcljx\" (UID: \"1b34e08e-81fb-4d17-8f15-2e24e94f47c0\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx" Oct 30 11:20:18.669360 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.669337 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf6n5\" (UniqueName: \"kubernetes.io/projected/1b34e08e-81fb-4d17-8f15-2e24e94f47c0-kube-api-access-kf6n5\") pod \"korrel8r-5cbcc89b8f-lcljx\" (UID: \"1b34e08e-81fb-4d17-8f15-2e24e94f47c0\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx" Oct 30 11:20:18.670037 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.670013 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b34e08e-81fb-4d17-8f15-2e24e94f47c0-config\") pod \"korrel8r-5cbcc89b8f-lcljx\" (UID: \"1b34e08e-81fb-4d17-8f15-2e24e94f47c0\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx" Oct 30 11:20:18.671633 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.671609 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1b34e08e-81fb-4d17-8f15-2e24e94f47c0-tls\") pod \"korrel8r-5cbcc89b8f-lcljx\" (UID: \"1b34e08e-81fb-4d17-8f15-2e24e94f47c0\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx" Oct 30 11:20:18.686553 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.686529 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" Oct 30 11:20:18.692302 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.692279 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf6n5\" (UniqueName: \"kubernetes.io/projected/1b34e08e-81fb-4d17-8f15-2e24e94f47c0-kube-api-access-kf6n5\") pod \"korrel8r-5cbcc89b8f-lcljx\" (UID: \"1b34e08e-81fb-4d17-8f15-2e24e94f47c0\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx" Oct 30 11:20:18.720413 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.720372 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx" Oct 30 11:20:18.927037 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.926955 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:18.942809 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.942783 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf"] Oct 30 11:20:18.943315 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:20:18.943285 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33defc9_4a83_4e81_9095_aa32e383792c.slice/crio-3cc136ee03058b9b673ee1118d9b479fe4db124661e40443e1cf657df0eb3c62 WatchSource:0}: Error finding container 3cc136ee03058b9b673ee1118d9b479fe4db124661e40443e1cf657df0eb3c62: Status 404 returned error can't find the container with id 3cc136ee03058b9b673ee1118d9b479fe4db124661e40443e1cf657df0eb3c62 Oct 30 11:20:18.971569 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.971541 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/22a70032-0628-4014-8cf5-aaac0fe64900-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2wfts\" (UID: \"22a70032-0628-4014-8cf5-aaac0fe64900\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" Oct 30 11:20:18.973767 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.973702 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/22a70032-0628-4014-8cf5-aaac0fe64900-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-2wfts\" (UID: \"22a70032-0628-4014-8cf5-aaac0fe64900\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" Oct 30 11:20:18.995643 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:20:18.995612 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode962d8b6_4549_4ec3_8e7e_bf003453f7a0.slice/crio-837e4342fb987e5630a91de2665b6d3feaf654a028c831151e9d60555684442a WatchSource:0}: Error finding container 837e4342fb987e5630a91de2665b6d3feaf654a028c831151e9d60555684442a: Status 404 returned error can't find the container with id 837e4342fb987e5630a91de2665b6d3feaf654a028c831151e9d60555684442a Oct 30 11:20:18.995857 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:18.995838 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd"] Oct 30 11:20:19.028804 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:19.028780 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx"] Oct 30 11:20:19.029070 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:20:19.029047 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b34e08e_81fb_4d17_8f15_2e24e94f47c0.slice/crio-49212920397ec540d4894c1a56fb9e7c33cb070cd3f093e4520ba76b91cf91b1 WatchSource:0}: Error finding container 49212920397ec540d4894c1a56fb9e7c33cb070cd3f093e4520ba76b91cf91b1: Status 404 returned error can't find the container with id 49212920397ec540d4894c1a56fb9e7c33cb070cd3f093e4520ba76b91cf91b1 Oct 30 11:20:19.096493 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:19.096467 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:19.193213 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:19.193186 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" Oct 30 11:20:19.274969 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:19.274941 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:19.494310 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:19.494276 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts"] Oct 30 11:20:19.882652 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:19.882540 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:19.918148 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:19.918096 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" event={"ID":"22a70032-0628-4014-8cf5-aaac0fe64900","Type":"ContainerStarted","Data":"33cedbf90600428b7d19d32bc747dd78e115d18c4d12676d76eb49177b301860"} Oct 30 11:20:19.920091 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:19.920063 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" event={"ID":"a33defc9-4a83-4e81-9095-aa32e383792c","Type":"ContainerStarted","Data":"ebfb21866551ae4032c89f5f23f4ce8fa7054712d322e20faea5bda9b1fdb57e"} Oct 30 11:20:19.920228 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:19.920130 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" event={"ID":"a33defc9-4a83-4e81-9095-aa32e383792c","Type":"ContainerStarted","Data":"549abeaf141117579caf86f961c41e4a8e9ec84e775ac34957b32011d3de6651"} Oct 30 11:20:19.920228 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:19.920170 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" event={"ID":"a33defc9-4a83-4e81-9095-aa32e383792c","Type":"ContainerStarted","Data":"82ca5403ace38c16289edd766983e820a58322f53c55fbfef4211ba7f952c3a8"} Oct 30 11:20:19.920228 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:19.920201 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" event={"ID":"a33defc9-4a83-4e81-9095-aa32e383792c","Type":"ContainerStarted","Data":"3cc136ee03058b9b673ee1118d9b479fe4db124661e40443e1cf657df0eb3c62"} Oct 30 11:20:19.921481 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:19.921456 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx" event={"ID":"1b34e08e-81fb-4d17-8f15-2e24e94f47c0","Type":"ContainerStarted","Data":"3130de7eddc09def3a888b18360ec5fa6b636163d69412222652304af4ffd13b"} Oct 30 11:20:19.921563 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:19.921503 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx" event={"ID":"1b34e08e-81fb-4d17-8f15-2e24e94f47c0","Type":"ContainerStarted","Data":"49212920397ec540d4894c1a56fb9e7c33cb070cd3f093e4520ba76b91cf91b1"} Oct 30 11:20:19.921563 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:19.921553 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx" Oct 30 11:20:19.923215 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:19.923196 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" event={"ID":"e962d8b6-4549-4ec3-8e7e-bf003453f7a0","Type":"ContainerStarted","Data":"7de94cd8314a221d0a4eddb716f354bfab2f272701ec737065523fee49ac6f1d"} Oct 30 11:20:19.923295 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:19.923239 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" event={"ID":"e962d8b6-4549-4ec3-8e7e-bf003453f7a0","Type":"ContainerStarted","Data":"c2704b6337a1d4056df4fd02608274c4e7b6f9745759466418469b66cd5aea53"} Oct 30 11:20:19.923295 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:19.923274 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" event={"ID":"e962d8b6-4549-4ec3-8e7e-bf003453f7a0","Type":"ContainerStarted","Data":"1791222d2e7cfe62c28cdc74f53766e585b66eb6553f736591ed8c0fbedbdeaf"} Oct 30 11:20:19.923362 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:19.923296 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" event={"ID":"e962d8b6-4549-4ec3-8e7e-bf003453f7a0","Type":"ContainerStarted","Data":"837e4342fb987e5630a91de2665b6d3feaf654a028c831151e9d60555684442a"} Oct 30 11:20:19.963543 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:19.963495 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" podStartSLOduration=1.96348167 podStartE2EDuration="1.96348167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:20:19.962855583 +0000 UTC m=+2110.464235068" watchObservedRunningTime="2026-10-30 11:20:19.96348167 +0000 UTC m=+2110.464861154" Oct 30 11:20:20.009157 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:20.009105 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" podStartSLOduration=2.009089909 podStartE2EDuration="2.009089909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:20:20.007863353 +0000 UTC m=+2110.509242838" watchObservedRunningTime="2026-10-30 11:20:20.009089909 +0000 UTC m=+2110.510469388" Oct 30 11:20:20.051211 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:20.051159 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx" podStartSLOduration=2.051146504 podStartE2EDuration="2.051146504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:20:20.049916907 +0000 UTC m=+2110.551296393" watchObservedRunningTime="2026-10-30 11:20:20.051146504 +0000 UTC m=+2110.552525988" Oct 30 11:20:20.057396 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:20.057359 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:20.929538 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:20.929496 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" event={"ID":"22a70032-0628-4014-8cf5-aaac0fe64900","Type":"ContainerStarted","Data":"9d06a0d05fa3800fd49de39549fd2cf3a5c4832756b4a5c264941154a1916e18"} Oct 30 11:20:20.930004 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:20.929902 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" Oct 30 11:20:20.970685 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:20.970633 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" podStartSLOduration=2.970620083 podStartE2EDuration="2.970620083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:20:20.969072201 +0000 UTC m=+2111.470451688" watchObservedRunningTime="2026-10-30 11:20:20.970620083 +0000 UTC m=+2111.471999568" Oct 30 11:20:22.512025 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:22.511992 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:22.750182 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:22.750141 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:22.924722 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:22.924624 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:23.149763 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.149735 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 11:20:23.153553 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.153531 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:20:23.153674 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.153659 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.158961 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.158935 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:23.169979 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.169954 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 11:20:23.171231 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.171206 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 11:20:23.173407 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.173388 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 11:20:23.174293 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.174272 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:23.200339 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.200310 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:20:23.238707 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.238675 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 11:20:23.238878 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.238858 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 11:20:23.238878 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.238872 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 11:20:23.239006 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.238873 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 11:20:23.239006 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.238868 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 11:20:23.239006 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.238904 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 11:20:23.239006 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.238899 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 11:20:23.239006 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.238923 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-4cc5x\"" Oct 30 11:20:23.239006 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.238942 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 11:20:23.239006 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.238982 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 11:20:23.239235 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.239042 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-w4zq4\"" Oct 30 11:20:23.270826 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.270802 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 11:20:23.310219 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.310188 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.310375 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.310233 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:23.310375 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.310264 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whx6j\" (UniqueName: \"kubernetes.io/projected/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-kube-api-access-whx6j\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:23.310375 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.310351 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17bb13d0-fa33-4e16-bf43-8ca7189291b8-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.310637 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.310387 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/17bb13d0-fa33-4e16-bf43-8ca7189291b8-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.310637 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.310500 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.310637 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.310531 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tv5r\" (UniqueName: \"kubernetes.io/projected/17bb13d0-fa33-4e16-bf43-8ca7189291b8-kube-api-access-9tv5r\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.310637 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.310608 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:23.310766 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.310648 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.310766 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.310679 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.310766 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.310710 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.310766 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.310753 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:23.310934 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.310769 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.310934 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.310826 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:23.310934 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.310896 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:23.310934 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.310915 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:23.310934 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.310929 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17bb13d0-fa33-4e16-bf43-8ca7189291b8-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.311140 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.310947 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/17bb13d0-fa33-4e16-bf43-8ca7189291b8-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.311140 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.310962 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/17bb13d0-fa33-4e16-bf43-8ca7189291b8-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.348919 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.348884 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:23.412146 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.412112 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:23.412146 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.412146 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.412341 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.412164 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:23.412341 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.412196 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:23.412341 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.412226 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:23.412341 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.412247 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17bb13d0-fa33-4e16-bf43-8ca7189291b8-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.412341 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.412291 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/17bb13d0-fa33-4e16-bf43-8ca7189291b8-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.412341 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.412316 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/17bb13d0-fa33-4e16-bf43-8ca7189291b8-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.412654 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.412403 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.412654 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.412463 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:23.412654 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.412488 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whx6j\" (UniqueName: \"kubernetes.io/projected/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-kube-api-access-whx6j\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:23.412982 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.412953 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/17bb13d0-fa33-4e16-bf43-8ca7189291b8-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.413141 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.413104 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17bb13d0-fa33-4e16-bf43-8ca7189291b8-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.413141 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.413054 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:23.413309 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.413151 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/17bb13d0-fa33-4e16-bf43-8ca7189291b8-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.413309 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.413187 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.413309 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.413212 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tv5r\" (UniqueName: \"kubernetes.io/projected/17bb13d0-fa33-4e16-bf43-8ca7189291b8-kube-api-access-9tv5r\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.413309 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.413258 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:23.413309 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.413282 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.416822 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.413308 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.416822 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.413336 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.416822 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.414031 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/17bb13d0-fa33-4e16-bf43-8ca7189291b8-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.416822 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.415564 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:23.416822 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.415804 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/17bb13d0-fa33-4e16-bf43-8ca7189291b8-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.416822 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.415949 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:23.416822 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.416069 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:23.416822 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.416741 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17bb13d0-fa33-4e16-bf43-8ca7189291b8-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.417215 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.416843 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.417628 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.417600 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.417948 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.417926 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17bb13d0-fa33-4e16-bf43-8ca7189291b8-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.418078 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.418050 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:23.418149 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.418083 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.418694 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.418666 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.418819 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.418792 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:23.419038 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.419020 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.419460 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.419413 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.434973 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.434923 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whx6j\" (UniqueName: \"kubernetes.io/projected/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-kube-api-access-whx6j\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:23.436315 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.436292 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tv5r\" (UniqueName: \"kubernetes.io/projected/17bb13d0-fa33-4e16-bf43-8ca7189291b8-kube-api-access-9tv5r\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:23.491565 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.491535 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:23.809281 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.809244 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:20:23.809631 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:20:23.809340 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67f9c21_0c37_4c52_92c3_6c31c9dc2d2e.slice/crio-3c2d81592a23d7b4bc133659dec176ad9b78931b118caca7e15e101ff4a24fff WatchSource:0}: Error finding container 3c2d81592a23d7b4bc133659dec176ad9b78931b118caca7e15e101ff4a24fff: Status 404 returned error can't find the container with id 3c2d81592a23d7b4bc133659dec176ad9b78931b118caca7e15e101ff4a24fff Oct 30 11:20:23.944454 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:23.944411 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e","Type":"ContainerStarted","Data":"3c2d81592a23d7b4bc133659dec176ad9b78931b118caca7e15e101ff4a24fff"} Oct 30 11:20:24.950107 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:24.950034 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e","Type":"ContainerStarted","Data":"d329c4449e57e94feaec69a0a7582ae3db7d3c035ad09cc8317d017219c36532"} Oct 30 11:20:26.257149 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:20:26.257112 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:20:26.936352 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:26.936318 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" Oct 30 11:20:27.349722 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:27.349692 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ae2fc6cc-1a29-41df-b109-2b8489d4e098\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:27.377243 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:27.377209 4289 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-ae2fc6cc-1a29-41df-b109-2b8489d4e098\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:27.450822 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:27.450787 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-ae2fc6cc-1a29-41df-b109-2b8489d4e098\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:27.450987 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:27.450937 4289 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-ae2fc6cc-1a29-41df-b109-2b8489d4e098\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:27.464784 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:27.464755 4289 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-ae2fc6cc-1a29-41df-b109-2b8489d4e098\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") DevicePath \"csi-41fb3b285706a9985249f9bffb659d03160e9e3e6288d3728f95a76a8535f583\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:27.567397 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:27.567365 4289 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-ae2fc6cc-1a29-41df-b109-2b8489d4e098\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/5f4e377c3e259c071aab429e31ca61846b879cc3476e2f02b70e888ccdcdb590/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:27.594572 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:27.594455 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-ae2fc6cc-1a29-41df-b109-2b8489d4e098\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:27.683488 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:27.683397 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:27.968496 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:27.968359 4289 generic.go:356] "Generic (PLEG): container finished" podID="d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e" containerID="d329c4449e57e94feaec69a0a7582ae3db7d3c035ad09cc8317d017219c36532" exitCode=0 Oct 30 11:20:27.968496 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:27.968402 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e","Type":"ContainerDied","Data":"d329c4449e57e94feaec69a0a7582ae3db7d3c035ad09cc8317d017219c36532"} Oct 30 11:20:28.004695 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:28.004667 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 11:20:28.008614 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:20:28.008587 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17bb13d0_fa33_4e16_bf43_8ca7189291b8.slice/crio-8b7e568319d555f78071f4c169eeed3cf77bc124966f04aa30e826a01073cc2a WatchSource:0}: Error finding container 8b7e568319d555f78071f4c169eeed3cf77bc124966f04aa30e826a01073cc2a: Status 404 returned error can't find the container with id 8b7e568319d555f78071f4c169eeed3cf77bc124966f04aa30e826a01073cc2a Oct 30 11:20:28.974570 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:28.974539 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"17bb13d0-fa33-4e16-bf43-8ca7189291b8","Type":"ContainerStarted","Data":"731d3b1901d6a398028250fe34008e65503edc76e1656158da8093ae3448b024"} Oct 30 11:20:28.974990 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:28.974612 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"17bb13d0-fa33-4e16-bf43-8ca7189291b8","Type":"ContainerStarted","Data":"8b7e568319d555f78071f4c169eeed3cf77bc124966f04aa30e826a01073cc2a"} Oct 30 11:20:28.976688 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:28.976667 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e","Type":"ContainerStarted","Data":"939084d548401ebe7f0f0c46fb8e897145291e00d948f86ff961c44c2c88c400"} Oct 30 11:20:28.976764 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:28.976718 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e","Type":"ContainerStarted","Data":"fffe9a0182189288cd2d58b2ec8df231662eafabfc285faba9de6bce7501fc8a"} Oct 30 11:20:28.976919 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:28.976906 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:29.084145 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:29.084094 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=2.599182122 podStartE2EDuration="6.084081165s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.484899043s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:20:29.08302483 +0000 UTC m=+2119.584404314" watchObservedRunningTime="2026-10-30 11:20:29.084081165 +0000 UTC m=+2119.585460650" Oct 30 11:20:30.932930 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:30.932899 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx" Oct 30 11:20:32.997216 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:32.997184 4289 generic.go:356] "Generic (PLEG): container finished" podID="17bb13d0-fa33-4e16-bf43-8ca7189291b8" containerID="731d3b1901d6a398028250fe34008e65503edc76e1656158da8093ae3448b024" exitCode=0 Oct 30 11:20:32.997588 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:32.997259 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"17bb13d0-fa33-4e16-bf43-8ca7189291b8","Type":"ContainerDied","Data":"731d3b1901d6a398028250fe34008e65503edc76e1656158da8093ae3448b024"} Oct 30 11:20:34.003900 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:34.003867 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"17bb13d0-fa33-4e16-bf43-8ca7189291b8","Type":"ContainerStarted","Data":"4574094aa55387e332d1ab1293e4e9a2bd93519893f9854c6d87b3305e809f96"} Oct 30 11:20:34.004260 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:34.003936 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"17bb13d0-fa33-4e16-bf43-8ca7189291b8","Type":"ContainerStarted","Data":"bf9e931a2ea2474f6e1100e2eac395dc2801a4d76d4b96f407ba9494afcbbe93"} Oct 30 11:20:34.004260 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:34.003961 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"17bb13d0-fa33-4e16-bf43-8ca7189291b8","Type":"ContainerStarted","Data":"b7ca7ed72ca49de380b8bde5ab25b98294177e6996e90c76afca8e25191302ba"} Oct 30 11:20:34.079827 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:34.079771 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=7.495102883 podStartE2EDuration="11.079755407s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.584652524s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:20:34.067356323 +0000 UTC m=+2124.568735811" watchObservedRunningTime="2026-10-30 11:20:34.079755407 +0000 UTC m=+2124.581134931" Oct 30 11:20:34.263655 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:34.263568 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:34.429567 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:34.429319 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:34.982442 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:34.982407 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:20:37.684317 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:37.684282 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:39.943340 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:39.943286 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:40.108508 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:40.108458 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:40.598865 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:40.598825 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:40.763825 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:40.763788 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:40.930844 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:40.930771 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:42.684587 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:42.684554 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:42.690205 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:42.690182 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:42.953529 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:42.953418 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:43.053014 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:43.052985 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:20:45.501985 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:45.501949 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:45.667536 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:45.667494 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:47.106943 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:47.106900 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:47.287029 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:47.286992 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:47.467171 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:47.467084 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:51.107527 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:51.107496 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:51.270249 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:51.270212 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:51.456638 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:51.456543 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:51.613903 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:51.613875 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:51.820445 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:51.820394 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:56.257599 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:20:56.257558 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:20:56.770595 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:56.770568 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:56.934848 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:56.934805 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:57.548218 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:57.548187 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:58.011996 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:58.011956 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:58.175107 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:58.175072 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:20:58.357692 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:20:58.357596 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:02.347226 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:02.347187 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:03.589180 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:03.589138 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:07.893522 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:07.893484 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:08.065067 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:08.065033 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:08.793846 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:08.793813 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:08.964297 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:08.964260 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:09.123934 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:09.123842 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:13.507594 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:13.507555 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:13.673041 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:13.673008 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:15.120115 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:15.120065 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:15.282106 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:15.282053 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:15.445753 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:15.445664 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:19.032059 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:19.032019 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:19.256619 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:19.256583 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:19.428413 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:19.428321 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:19.595442 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:19.595402 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:19.759252 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:19.759215 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:20.842568 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:20.842535 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:21.000905 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:21.000862 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:24.774100 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:24.774064 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:24.946963 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:24.946930 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:26.257710 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:21:26.257675 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:21:27.250206 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.250164 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts"] Oct 30 11:21:27.250546 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.250453 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" podUID="22a70032-0628-4014-8cf5-aaac0fe64900" containerName="kube-rbac-proxy" containerID="cri-o://9d06a0d05fa3800fd49de39549fd2cf3a5c4832756b4a5c264941154a1916e18" gracePeriod=30 Oct 30 11:21:27.280155 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.280127 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf"] Oct 30 11:21:27.280570 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.280445 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" podUID="a33defc9-4a83-4e81-9095-aa32e383792c" containerName="kube-rbac-proxy" containerID="cri-o://82ca5403ace38c16289edd766983e820a58322f53c55fbfef4211ba7f952c3a8" gracePeriod=30 Oct 30 11:21:27.280570 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.280462 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" podUID="a33defc9-4a83-4e81-9095-aa32e383792c" containerName="prom-label-proxy" containerID="cri-o://ebfb21866551ae4032c89f5f23f4ce8fa7054712d322e20faea5bda9b1fdb57e" gracePeriod=30 Oct 30 11:21:27.280570 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.280484 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" podUID="a33defc9-4a83-4e81-9095-aa32e383792c" containerName="method-gate" containerID="cri-o://549abeaf141117579caf86f961c41e4a8e9ec84e775ac34957b32011d3de6651" gracePeriod=30 Oct 30 11:21:27.290565 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.290543 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd"] Oct 30 11:21:27.290758 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.290737 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" podUID="e962d8b6-4549-4ec3-8e7e-bf003453f7a0" containerName="kube-rbac-proxy" containerID="cri-o://1791222d2e7cfe62c28cdc74f53766e585b66eb6553f736591ed8c0fbedbdeaf" gracePeriod=30 Oct 30 11:21:27.290829 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.290760 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" podUID="e962d8b6-4549-4ec3-8e7e-bf003453f7a0" containerName="prom-label-proxy" containerID="cri-o://7de94cd8314a221d0a4eddb716f354bfab2f272701ec737065523fee49ac6f1d" gracePeriod=30 Oct 30 11:21:27.290829 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.290779 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" podUID="e962d8b6-4549-4ec3-8e7e-bf003453f7a0" containerName="method-gate" containerID="cri-o://c2704b6337a1d4056df4fd02608274c4e7b6f9745759466418469b66cd5aea53" gracePeriod=30 Oct 30 11:21:27.303600 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.303570 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx"] Oct 30 11:21:27.303749 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.303732 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx" podUID="1b34e08e-81fb-4d17-8f15-2e24e94f47c0" containerName="korrel8r" containerID="cri-o://3130de7eddc09def3a888b18360ec5fa6b636163d69412222652304af4ffd13b" gracePeriod=30 Oct 30 11:21:27.567249 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.567007 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" Oct 30 11:21:27.600678 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.599120 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx" Oct 30 11:21:27.655372 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.655345 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" Oct 30 11:21:27.657571 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.657551 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" Oct 30 11:21:27.696306 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.696268 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:27.697594 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.697570 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a70032-0628-4014-8cf5-aaac0fe64900-kube-api-access-mljts\" (UniqueName: \"kubernetes.io/projected/22a70032-0628-4014-8cf5-aaac0fe64900-kube-api-access-mljts\") pod \"22a70032-0628-4014-8cf5-aaac0fe64900\" (UID: \"22a70032-0628-4014-8cf5-aaac0fe64900\") " Oct 30 11:21:27.697728 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.697614 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b34e08e-81fb-4d17-8f15-2e24e94f47c0-config\" (UniqueName: \"kubernetes.io/configmap/1b34e08e-81fb-4d17-8f15-2e24e94f47c0-config\") pod \"1b34e08e-81fb-4d17-8f15-2e24e94f47c0\" (UID: \"1b34e08e-81fb-4d17-8f15-2e24e94f47c0\") " Oct 30 11:21:27.697728 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.697652 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22a70032-0628-4014-8cf5-aaac0fe64900-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/22a70032-0628-4014-8cf5-aaac0fe64900-prometheus-client-cert\") pod \"22a70032-0628-4014-8cf5-aaac0fe64900\" (UID: \"22a70032-0628-4014-8cf5-aaac0fe64900\") " Oct 30 11:21:27.697728 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.697693 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22a70032-0628-4014-8cf5-aaac0fe64900-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/22a70032-0628-4014-8cf5-aaac0fe64900-prometheus-ca\") pod \"22a70032-0628-4014-8cf5-aaac0fe64900\" (UID: \"22a70032-0628-4014-8cf5-aaac0fe64900\") " Oct 30 11:21:27.697890 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.697791 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22a70032-0628-4014-8cf5-aaac0fe64900-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22a70032-0628-4014-8cf5-aaac0fe64900-kube-rbac-proxy-config\") pod \"22a70032-0628-4014-8cf5-aaac0fe64900\" (UID: \"22a70032-0628-4014-8cf5-aaac0fe64900\") " Oct 30 11:21:27.697890 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.697825 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b34e08e-81fb-4d17-8f15-2e24e94f47c0-tls\" (UniqueName: \"kubernetes.io/secret/1b34e08e-81fb-4d17-8f15-2e24e94f47c0-tls\") pod \"1b34e08e-81fb-4d17-8f15-2e24e94f47c0\" (UID: \"1b34e08e-81fb-4d17-8f15-2e24e94f47c0\") " Oct 30 11:21:27.697890 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.697855 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b34e08e-81fb-4d17-8f15-2e24e94f47c0-kube-api-access-kf6n5\" (UniqueName: \"kubernetes.io/projected/1b34e08e-81fb-4d17-8f15-2e24e94f47c0-kube-api-access-kf6n5\") pod \"1b34e08e-81fb-4d17-8f15-2e24e94f47c0\" (UID: \"1b34e08e-81fb-4d17-8f15-2e24e94f47c0\") " Oct 30 11:21:27.698047 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.697917 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22a70032-0628-4014-8cf5-aaac0fe64900-tls\" (UniqueName: \"kubernetes.io/secret/22a70032-0628-4014-8cf5-aaac0fe64900-tls\") pod \"22a70032-0628-4014-8cf5-aaac0fe64900\" (UID: \"22a70032-0628-4014-8cf5-aaac0fe64900\") " Oct 30 11:21:27.698589 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.698241 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a70032-0628-4014-8cf5-aaac0fe64900-prometheus-ca" pod "22a70032-0628-4014-8cf5-aaac0fe64900" (UID: "22a70032-0628-4014-8cf5-aaac0fe64900"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:21:27.698937 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.698895 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b34e08e-81fb-4d17-8f15-2e24e94f47c0-config" pod "1b34e08e-81fb-4d17-8f15-2e24e94f47c0" (UID: "1b34e08e-81fb-4d17-8f15-2e24e94f47c0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:21:27.701275 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.701246 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a70032-0628-4014-8cf5-aaac0fe64900-tls" pod "22a70032-0628-4014-8cf5-aaac0fe64900" (UID: "22a70032-0628-4014-8cf5-aaac0fe64900"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:21:27.703377 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.703338 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b34e08e-81fb-4d17-8f15-2e24e94f47c0-kube-api-access-kf6n5" pod "1b34e08e-81fb-4d17-8f15-2e24e94f47c0" (UID: "1b34e08e-81fb-4d17-8f15-2e24e94f47c0"). InnerVolumeSpecName "kube-api-access-kf6n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:21:27.703501 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.703474 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a70032-0628-4014-8cf5-aaac0fe64900-kube-api-access-mljts" pod "22a70032-0628-4014-8cf5-aaac0fe64900" (UID: "22a70032-0628-4014-8cf5-aaac0fe64900"). InnerVolumeSpecName "kube-api-access-mljts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:21:27.704520 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.703855 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a70032-0628-4014-8cf5-aaac0fe64900-prometheus-client-cert" pod "22a70032-0628-4014-8cf5-aaac0fe64900" (UID: "22a70032-0628-4014-8cf5-aaac0fe64900"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:21:27.704520 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.703953 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a70032-0628-4014-8cf5-aaac0fe64900-kube-rbac-proxy-config" pod "22a70032-0628-4014-8cf5-aaac0fe64900" (UID: "22a70032-0628-4014-8cf5-aaac0fe64900"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:21:27.704520 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.704111 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b34e08e-81fb-4d17-8f15-2e24e94f47c0-tls" pod "1b34e08e-81fb-4d17-8f15-2e24e94f47c0" (UID: "1b34e08e-81fb-4d17-8f15-2e24e94f47c0"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:21:27.758616 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.758580 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:21:27.758841 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.758818 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e" containerName="alertmanager" containerID="cri-o://fffe9a0182189288cd2d58b2ec8df231662eafabfc285faba9de6bce7501fc8a" gracePeriod=120 Oct 30 11:21:27.758914 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.758865 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e" containerName="config-reloader" containerID="cri-o://939084d548401ebe7f0f0c46fb8e897145291e00d948f86ff961c44c2c88c400" gracePeriod=120 Oct 30 11:21:27.775252 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.775226 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 11:21:27.780681 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.780645 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="17bb13d0-fa33-4e16-bf43-8ca7189291b8" containerName="prometheus" containerID="cri-o://b7ca7ed72ca49de380b8bde5ab25b98294177e6996e90c76afca8e25191302ba" gracePeriod=600 Oct 30 11:21:27.780809 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.780654 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="17bb13d0-fa33-4e16-bf43-8ca7189291b8" containerName="config-reloader" containerID="cri-o://bf9e931a2ea2474f6e1100e2eac395dc2801a4d76d4b96f407ba9494afcbbe93" gracePeriod=600 Oct 30 11:21:27.780883 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.780651 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="17bb13d0-fa33-4e16-bf43-8ca7189291b8" containerName="thanos-sidecar" containerID="cri-o://4574094aa55387e332d1ab1293e4e9a2bd93519893f9854c6d87b3305e809f96" gracePeriod=600 Oct 30 11:21:27.798612 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.798590 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a33defc9-4a83-4e81-9095-aa32e383792c-client-ca\" (UniqueName: \"kubernetes.io/configmap/a33defc9-4a83-4e81-9095-aa32e383792c-client-ca\") pod \"a33defc9-4a83-4e81-9095-aa32e383792c\" (UID: \"a33defc9-4a83-4e81-9095-aa32e383792c\") " Oct 30 11:21:27.798715 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.798640 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e962d8b6-4549-4ec3-8e7e-bf003453f7a0-kube-api-access-6prsv\" (UniqueName: \"kubernetes.io/projected/e962d8b6-4549-4ec3-8e7e-bf003453f7a0-kube-api-access-6prsv\") pod \"e962d8b6-4549-4ec3-8e7e-bf003453f7a0\" (UID: \"e962d8b6-4549-4ec3-8e7e-bf003453f7a0\") " Oct 30 11:21:27.798715 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.798679 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a33defc9-4a83-4e81-9095-aa32e383792c-tls\" (UniqueName: \"kubernetes.io/secret/a33defc9-4a83-4e81-9095-aa32e383792c-tls\") pod \"a33defc9-4a83-4e81-9095-aa32e383792c\" (UID: \"a33defc9-4a83-4e81-9095-aa32e383792c\") " Oct 30 11:21:27.798840 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.798719 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e962d8b6-4549-4ec3-8e7e-bf003453f7a0-tls\" (UniqueName: \"kubernetes.io/secret/e962d8b6-4549-4ec3-8e7e-bf003453f7a0-tls\") pod \"e962d8b6-4549-4ec3-8e7e-bf003453f7a0\" (UID: \"e962d8b6-4549-4ec3-8e7e-bf003453f7a0\") " Oct 30 11:21:27.798840 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.798751 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a33defc9-4a83-4e81-9095-aa32e383792c-kube-api-access-t76pk\" (UniqueName: \"kubernetes.io/projected/a33defc9-4a83-4e81-9095-aa32e383792c-kube-api-access-t76pk\") pod \"a33defc9-4a83-4e81-9095-aa32e383792c\" (UID: \"a33defc9-4a83-4e81-9095-aa32e383792c\") " Oct 30 11:21:27.798840 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.798780 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a33defc9-4a83-4e81-9095-aa32e383792c-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/a33defc9-4a83-4e81-9095-aa32e383792c-prometheus-tls-ca\") pod \"a33defc9-4a83-4e81-9095-aa32e383792c\" (UID: \"a33defc9-4a83-4e81-9095-aa32e383792c\") " Oct 30 11:21:27.799073 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.798864 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a33defc9-4a83-4e81-9095-aa32e383792c-config\" (UniqueName: \"kubernetes.io/configmap/a33defc9-4a83-4e81-9095-aa32e383792c-config\") pod \"a33defc9-4a83-4e81-9095-aa32e383792c\" (UID: \"a33defc9-4a83-4e81-9095-aa32e383792c\") " Oct 30 11:21:27.799073 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.798906 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e962d8b6-4549-4ec3-8e7e-bf003453f7a0-config\" (UniqueName: \"kubernetes.io/configmap/e962d8b6-4549-4ec3-8e7e-bf003453f7a0-config\") pod \"e962d8b6-4549-4ec3-8e7e-bf003453f7a0\" (UID: \"e962d8b6-4549-4ec3-8e7e-bf003453f7a0\") " Oct 30 11:21:27.799211 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.799194 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mljts\" (UniqueName: \"kubernetes.io/projected/22a70032-0628-4014-8cf5-aaac0fe64900-kube-api-access-mljts\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:27.799271 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.799217 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b34e08e-81fb-4d17-8f15-2e24e94f47c0-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:27.799271 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.799232 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/22a70032-0628-4014-8cf5-aaac0fe64900-prometheus-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:27.799271 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.799246 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/22a70032-0628-4014-8cf5-aaac0fe64900-prometheus-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:27.799443 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.799391 4289 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22a70032-0628-4014-8cf5-aaac0fe64900-kube-rbac-proxy-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:27.799443 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.799412 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1b34e08e-81fb-4d17-8f15-2e24e94f47c0-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:27.799562 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.799444 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf6n5\" (UniqueName: \"kubernetes.io/projected/1b34e08e-81fb-4d17-8f15-2e24e94f47c0-kube-api-access-kf6n5\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:27.799562 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.799458 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/22a70032-0628-4014-8cf5-aaac0fe64900-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:27.799766 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.799747 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e962d8b6-4549-4ec3-8e7e-bf003453f7a0-config" pod "e962d8b6-4549-4ec3-8e7e-bf003453f7a0" (UID: "e962d8b6-4549-4ec3-8e7e-bf003453f7a0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:21:27.799873 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.799849 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33defc9-4a83-4e81-9095-aa32e383792c-prometheus-tls-ca" pod "a33defc9-4a83-4e81-9095-aa32e383792c" (UID: "a33defc9-4a83-4e81-9095-aa32e383792c"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:21:27.800092 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.799607 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33defc9-4a83-4e81-9095-aa32e383792c-config" pod "a33defc9-4a83-4e81-9095-aa32e383792c" (UID: "a33defc9-4a83-4e81-9095-aa32e383792c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:21:27.800200 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.800179 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33defc9-4a83-4e81-9095-aa32e383792c-client-ca" pod "a33defc9-4a83-4e81-9095-aa32e383792c" (UID: "a33defc9-4a83-4e81-9095-aa32e383792c"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:21:27.801080 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.801049 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e962d8b6-4549-4ec3-8e7e-bf003453f7a0-tls" pod "e962d8b6-4549-4ec3-8e7e-bf003453f7a0" (UID: "e962d8b6-4549-4ec3-8e7e-bf003453f7a0"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:21:27.801180 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.801131 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a33defc9-4a83-4e81-9095-aa32e383792c-tls" pod "a33defc9-4a83-4e81-9095-aa32e383792c" (UID: "a33defc9-4a83-4e81-9095-aa32e383792c"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:21:27.801805 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.801787 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33defc9-4a83-4e81-9095-aa32e383792c-kube-api-access-t76pk" pod "a33defc9-4a83-4e81-9095-aa32e383792c" (UID: "a33defc9-4a83-4e81-9095-aa32e383792c"). InnerVolumeSpecName "kube-api-access-t76pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:21:27.801894 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.801882 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e962d8b6-4549-4ec3-8e7e-bf003453f7a0-kube-api-access-6prsv" pod "e962d8b6-4549-4ec3-8e7e-bf003453f7a0" (UID: "e962d8b6-4549-4ec3-8e7e-bf003453f7a0"). InnerVolumeSpecName "kube-api-access-6prsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:21:27.875451 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.875407 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:27.900284 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.900255 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e962d8b6-4549-4ec3-8e7e-bf003453f7a0-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:27.900284 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.900282 4289 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/a33defc9-4a83-4e81-9095-aa32e383792c-client-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:27.900504 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.900295 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6prsv\" (UniqueName: \"kubernetes.io/projected/e962d8b6-4549-4ec3-8e7e-bf003453f7a0-kube-api-access-6prsv\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:27.900504 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.900304 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a33defc9-4a83-4e81-9095-aa32e383792c-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:27.900504 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.900313 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e962d8b6-4549-4ec3-8e7e-bf003453f7a0-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:27.900504 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.900321 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t76pk\" (UniqueName: \"kubernetes.io/projected/a33defc9-4a83-4e81-9095-aa32e383792c-kube-api-access-t76pk\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:27.900504 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.900328 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/a33defc9-4a83-4e81-9095-aa32e383792c-prometheus-tls-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:27.900504 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:27.900336 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a33defc9-4a83-4e81-9095-aa32e383792c-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:28.047038 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.046956 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="17bb13d0-fa33-4e16-bf43-8ca7189291b8" containerName="prometheus" probeResult="failure" output="Get \"https://10.134.0.85:9090/-/ready\": dial tcp 10.134.0.85:9090: connect: connection refused" Oct 30 11:21:28.163960 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.163928 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:28.253470 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.253408 4289 generic.go:356] "Generic (PLEG): container finished" podID="1b34e08e-81fb-4d17-8f15-2e24e94f47c0" containerID="3130de7eddc09def3a888b18360ec5fa6b636163d69412222652304af4ffd13b" exitCode=0 Oct 30 11:21:28.253632 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.253489 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx" Oct 30 11:21:28.253632 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.253487 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx" event={"ID":"1b34e08e-81fb-4d17-8f15-2e24e94f47c0","Type":"ContainerDied","Data":"3130de7eddc09def3a888b18360ec5fa6b636163d69412222652304af4ffd13b"} Oct 30 11:21:28.253752 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.253672 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx" event={"ID":"1b34e08e-81fb-4d17-8f15-2e24e94f47c0","Type":"ContainerDied","Data":"49212920397ec540d4894c1a56fb9e7c33cb070cd3f093e4520ba76b91cf91b1"} Oct 30 11:21:28.253752 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.253703 4289 scope.go:122] "RemoveContainer" containerID="3130de7eddc09def3a888b18360ec5fa6b636163d69412222652304af4ffd13b" Oct 30 11:21:28.255773 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.255752 4289 generic.go:356] "Generic (PLEG): container finished" podID="e962d8b6-4549-4ec3-8e7e-bf003453f7a0" containerID="7de94cd8314a221d0a4eddb716f354bfab2f272701ec737065523fee49ac6f1d" exitCode=0 Oct 30 11:21:28.255773 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.255768 4289 generic.go:356] "Generic (PLEG): container finished" podID="e962d8b6-4549-4ec3-8e7e-bf003453f7a0" containerID="c2704b6337a1d4056df4fd02608274c4e7b6f9745759466418469b66cd5aea53" exitCode=0 Oct 30 11:21:28.255773 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.255776 4289 generic.go:356] "Generic (PLEG): container finished" podID="e962d8b6-4549-4ec3-8e7e-bf003453f7a0" containerID="1791222d2e7cfe62c28cdc74f53766e585b66eb6553f736591ed8c0fbedbdeaf" exitCode=0 Oct 30 11:21:28.256023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.255785 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" event={"ID":"e962d8b6-4549-4ec3-8e7e-bf003453f7a0","Type":"ContainerDied","Data":"7de94cd8314a221d0a4eddb716f354bfab2f272701ec737065523fee49ac6f1d"} Oct 30 11:21:28.256023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.255857 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" Oct 30 11:21:28.256023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.255863 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" event={"ID":"e962d8b6-4549-4ec3-8e7e-bf003453f7a0","Type":"ContainerDied","Data":"c2704b6337a1d4056df4fd02608274c4e7b6f9745759466418469b66cd5aea53"} Oct 30 11:21:28.256023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.255909 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" event={"ID":"e962d8b6-4549-4ec3-8e7e-bf003453f7a0","Type":"ContainerDied","Data":"1791222d2e7cfe62c28cdc74f53766e585b66eb6553f736591ed8c0fbedbdeaf"} Oct 30 11:21:28.256023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.255944 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd" event={"ID":"e962d8b6-4549-4ec3-8e7e-bf003453f7a0","Type":"ContainerDied","Data":"837e4342fb987e5630a91de2665b6d3feaf654a028c831151e9d60555684442a"} Oct 30 11:21:28.258311 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.258288 4289 generic.go:356] "Generic (PLEG): container finished" podID="17bb13d0-fa33-4e16-bf43-8ca7189291b8" containerID="4574094aa55387e332d1ab1293e4e9a2bd93519893f9854c6d87b3305e809f96" exitCode=0 Oct 30 11:21:28.258421 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.258312 4289 generic.go:356] "Generic (PLEG): container finished" podID="17bb13d0-fa33-4e16-bf43-8ca7189291b8" containerID="bf9e931a2ea2474f6e1100e2eac395dc2801a4d76d4b96f407ba9494afcbbe93" exitCode=0 Oct 30 11:21:28.258421 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.258323 4289 generic.go:356] "Generic (PLEG): container finished" podID="17bb13d0-fa33-4e16-bf43-8ca7189291b8" containerID="b7ca7ed72ca49de380b8bde5ab25b98294177e6996e90c76afca8e25191302ba" exitCode=0 Oct 30 11:21:28.258421 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.258407 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"17bb13d0-fa33-4e16-bf43-8ca7189291b8","Type":"ContainerDied","Data":"4574094aa55387e332d1ab1293e4e9a2bd93519893f9854c6d87b3305e809f96"} Oct 30 11:21:28.258602 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.258489 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"17bb13d0-fa33-4e16-bf43-8ca7189291b8","Type":"ContainerDied","Data":"bf9e931a2ea2474f6e1100e2eac395dc2801a4d76d4b96f407ba9494afcbbe93"} Oct 30 11:21:28.258602 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.258525 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"17bb13d0-fa33-4e16-bf43-8ca7189291b8","Type":"ContainerDied","Data":"b7ca7ed72ca49de380b8bde5ab25b98294177e6996e90c76afca8e25191302ba"} Oct 30 11:21:28.259782 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.259757 4289 generic.go:356] "Generic (PLEG): container finished" podID="22a70032-0628-4014-8cf5-aaac0fe64900" containerID="9d06a0d05fa3800fd49de39549fd2cf3a5c4832756b4a5c264941154a1916e18" exitCode=0 Oct 30 11:21:28.259906 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.259831 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" Oct 30 11:21:28.260170 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.259842 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" event={"ID":"22a70032-0628-4014-8cf5-aaac0fe64900","Type":"ContainerDied","Data":"9d06a0d05fa3800fd49de39549fd2cf3a5c4832756b4a5c264941154a1916e18"} Oct 30 11:21:28.260170 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.260039 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts" event={"ID":"22a70032-0628-4014-8cf5-aaac0fe64900","Type":"ContainerDied","Data":"33cedbf90600428b7d19d32bc747dd78e115d18c4d12676d76eb49177b301860"} Oct 30 11:21:28.262256 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.262237 4289 generic.go:356] "Generic (PLEG): container finished" podID="d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e" containerID="939084d548401ebe7f0f0c46fb8e897145291e00d948f86ff961c44c2c88c400" exitCode=0 Oct 30 11:21:28.262367 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.262311 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e","Type":"ContainerDied","Data":"939084d548401ebe7f0f0c46fb8e897145291e00d948f86ff961c44c2c88c400"} Oct 30 11:21:28.264202 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.264182 4289 generic.go:356] "Generic (PLEG): container finished" podID="a33defc9-4a83-4e81-9095-aa32e383792c" containerID="ebfb21866551ae4032c89f5f23f4ce8fa7054712d322e20faea5bda9b1fdb57e" exitCode=0 Oct 30 11:21:28.264202 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.264198 4289 generic.go:356] "Generic (PLEG): container finished" podID="a33defc9-4a83-4e81-9095-aa32e383792c" containerID="549abeaf141117579caf86f961c41e4a8e9ec84e775ac34957b32011d3de6651" exitCode=0 Oct 30 11:21:28.264202 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.264204 4289 generic.go:356] "Generic (PLEG): container finished" podID="a33defc9-4a83-4e81-9095-aa32e383792c" containerID="82ca5403ace38c16289edd766983e820a58322f53c55fbfef4211ba7f952c3a8" exitCode=0 Oct 30 11:21:28.264408 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.264233 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" event={"ID":"a33defc9-4a83-4e81-9095-aa32e383792c","Type":"ContainerDied","Data":"ebfb21866551ae4032c89f5f23f4ce8fa7054712d322e20faea5bda9b1fdb57e"} Oct 30 11:21:28.264408 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.264269 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" event={"ID":"a33defc9-4a83-4e81-9095-aa32e383792c","Type":"ContainerDied","Data":"549abeaf141117579caf86f961c41e4a8e9ec84e775ac34957b32011d3de6651"} Oct 30 11:21:28.264408 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.264293 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" event={"ID":"a33defc9-4a83-4e81-9095-aa32e383792c","Type":"ContainerDied","Data":"82ca5403ace38c16289edd766983e820a58322f53c55fbfef4211ba7f952c3a8"} Oct 30 11:21:28.264408 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.264314 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" event={"ID":"a33defc9-4a83-4e81-9095-aa32e383792c","Type":"ContainerDied","Data":"3cc136ee03058b9b673ee1118d9b479fe4db124661e40443e1cf657df0eb3c62"} Oct 30 11:21:28.264408 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.264384 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf" Oct 30 11:21:28.285628 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.285611 4289 scope.go:122] "RemoveContainer" containerID="3130de7eddc09def3a888b18360ec5fa6b636163d69412222652304af4ffd13b" Oct 30 11:21:28.285989 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:21:28.285912 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3130de7eddc09def3a888b18360ec5fa6b636163d69412222652304af4ffd13b\": container with ID starting with 3130de7eddc09def3a888b18360ec5fa6b636163d69412222652304af4ffd13b not found: ID does not exist" containerID="3130de7eddc09def3a888b18360ec5fa6b636163d69412222652304af4ffd13b" Oct 30 11:21:28.285989 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.285944 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3130de7eddc09def3a888b18360ec5fa6b636163d69412222652304af4ffd13b"} err="failed to get container status \"3130de7eddc09def3a888b18360ec5fa6b636163d69412222652304af4ffd13b\": rpc error: code = NotFound desc = could not find container \"3130de7eddc09def3a888b18360ec5fa6b636163d69412222652304af4ffd13b\": container with ID starting with 3130de7eddc09def3a888b18360ec5fa6b636163d69412222652304af4ffd13b not found: ID does not exist" Oct 30 11:21:28.285989 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.285961 4289 scope.go:122] "RemoveContainer" containerID="7de94cd8314a221d0a4eddb716f354bfab2f272701ec737065523fee49ac6f1d" Oct 30 11:21:28.293586 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.293569 4289 scope.go:122] "RemoveContainer" containerID="c2704b6337a1d4056df4fd02608274c4e7b6f9745759466418469b66cd5aea53" Oct 30 11:21:28.300746 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.300728 4289 scope.go:122] "RemoveContainer" containerID="1791222d2e7cfe62c28cdc74f53766e585b66eb6553f736591ed8c0fbedbdeaf" Oct 30 11:21:28.307536 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.307521 4289 scope.go:122] "RemoveContainer" containerID="7de94cd8314a221d0a4eddb716f354bfab2f272701ec737065523fee49ac6f1d" Oct 30 11:21:28.307813 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:21:28.307792 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7de94cd8314a221d0a4eddb716f354bfab2f272701ec737065523fee49ac6f1d\": container with ID starting with 7de94cd8314a221d0a4eddb716f354bfab2f272701ec737065523fee49ac6f1d not found: ID does not exist" containerID="7de94cd8314a221d0a4eddb716f354bfab2f272701ec737065523fee49ac6f1d" Oct 30 11:21:28.307896 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.307823 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de94cd8314a221d0a4eddb716f354bfab2f272701ec737065523fee49ac6f1d"} err="failed to get container status \"7de94cd8314a221d0a4eddb716f354bfab2f272701ec737065523fee49ac6f1d\": rpc error: code = NotFound desc = could not find container \"7de94cd8314a221d0a4eddb716f354bfab2f272701ec737065523fee49ac6f1d\": container with ID starting with 7de94cd8314a221d0a4eddb716f354bfab2f272701ec737065523fee49ac6f1d not found: ID does not exist" Oct 30 11:21:28.307896 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.307848 4289 scope.go:122] "RemoveContainer" containerID="c2704b6337a1d4056df4fd02608274c4e7b6f9745759466418469b66cd5aea53" Oct 30 11:21:28.308108 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:21:28.308094 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2704b6337a1d4056df4fd02608274c4e7b6f9745759466418469b66cd5aea53\": container with ID starting with c2704b6337a1d4056df4fd02608274c4e7b6f9745759466418469b66cd5aea53 not found: ID does not exist" containerID="c2704b6337a1d4056df4fd02608274c4e7b6f9745759466418469b66cd5aea53" Oct 30 11:21:28.308152 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.308111 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2704b6337a1d4056df4fd02608274c4e7b6f9745759466418469b66cd5aea53"} err="failed to get container status \"c2704b6337a1d4056df4fd02608274c4e7b6f9745759466418469b66cd5aea53\": rpc error: code = NotFound desc = could not find container \"c2704b6337a1d4056df4fd02608274c4e7b6f9745759466418469b66cd5aea53\": container with ID starting with c2704b6337a1d4056df4fd02608274c4e7b6f9745759466418469b66cd5aea53 not found: ID does not exist" Oct 30 11:21:28.308152 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.308123 4289 scope.go:122] "RemoveContainer" containerID="1791222d2e7cfe62c28cdc74f53766e585b66eb6553f736591ed8c0fbedbdeaf" Oct 30 11:21:28.308367 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:21:28.308344 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1791222d2e7cfe62c28cdc74f53766e585b66eb6553f736591ed8c0fbedbdeaf\": container with ID starting with 1791222d2e7cfe62c28cdc74f53766e585b66eb6553f736591ed8c0fbedbdeaf not found: ID does not exist" containerID="1791222d2e7cfe62c28cdc74f53766e585b66eb6553f736591ed8c0fbedbdeaf" Oct 30 11:21:28.308403 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.308372 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1791222d2e7cfe62c28cdc74f53766e585b66eb6553f736591ed8c0fbedbdeaf"} err="failed to get container status \"1791222d2e7cfe62c28cdc74f53766e585b66eb6553f736591ed8c0fbedbdeaf\": rpc error: code = NotFound desc = could not find container \"1791222d2e7cfe62c28cdc74f53766e585b66eb6553f736591ed8c0fbedbdeaf\": container with ID starting with 1791222d2e7cfe62c28cdc74f53766e585b66eb6553f736591ed8c0fbedbdeaf not found: ID does not exist" Oct 30 11:21:28.308403 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.308387 4289 scope.go:122] "RemoveContainer" containerID="7de94cd8314a221d0a4eddb716f354bfab2f272701ec737065523fee49ac6f1d" Oct 30 11:21:28.308817 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.308799 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de94cd8314a221d0a4eddb716f354bfab2f272701ec737065523fee49ac6f1d"} err="failed to get container status \"7de94cd8314a221d0a4eddb716f354bfab2f272701ec737065523fee49ac6f1d\": rpc error: code = NotFound desc = could not find container \"7de94cd8314a221d0a4eddb716f354bfab2f272701ec737065523fee49ac6f1d\": container with ID starting with 7de94cd8314a221d0a4eddb716f354bfab2f272701ec737065523fee49ac6f1d not found: ID does not exist" Oct 30 11:21:28.308863 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.308818 4289 scope.go:122] "RemoveContainer" containerID="c2704b6337a1d4056df4fd02608274c4e7b6f9745759466418469b66cd5aea53" Oct 30 11:21:28.309011 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.308991 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2704b6337a1d4056df4fd02608274c4e7b6f9745759466418469b66cd5aea53"} err="failed to get container status \"c2704b6337a1d4056df4fd02608274c4e7b6f9745759466418469b66cd5aea53\": rpc error: code = NotFound desc = could not find container \"c2704b6337a1d4056df4fd02608274c4e7b6f9745759466418469b66cd5aea53\": container with ID starting with c2704b6337a1d4056df4fd02608274c4e7b6f9745759466418469b66cd5aea53 not found: ID does not exist" Oct 30 11:21:28.309054 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.309010 4289 scope.go:122] "RemoveContainer" containerID="1791222d2e7cfe62c28cdc74f53766e585b66eb6553f736591ed8c0fbedbdeaf" Oct 30 11:21:28.309261 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.309232 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1791222d2e7cfe62c28cdc74f53766e585b66eb6553f736591ed8c0fbedbdeaf"} err="failed to get container status \"1791222d2e7cfe62c28cdc74f53766e585b66eb6553f736591ed8c0fbedbdeaf\": rpc error: code = NotFound desc = could not find container \"1791222d2e7cfe62c28cdc74f53766e585b66eb6553f736591ed8c0fbedbdeaf\": container with ID starting with 1791222d2e7cfe62c28cdc74f53766e585b66eb6553f736591ed8c0fbedbdeaf not found: ID does not exist" Oct 30 11:21:28.309261 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.309261 4289 scope.go:122] "RemoveContainer" containerID="7de94cd8314a221d0a4eddb716f354bfab2f272701ec737065523fee49ac6f1d" Oct 30 11:21:28.309502 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.309479 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de94cd8314a221d0a4eddb716f354bfab2f272701ec737065523fee49ac6f1d"} err="failed to get container status \"7de94cd8314a221d0a4eddb716f354bfab2f272701ec737065523fee49ac6f1d\": rpc error: code = NotFound desc = could not find container \"7de94cd8314a221d0a4eddb716f354bfab2f272701ec737065523fee49ac6f1d\": container with ID starting with 7de94cd8314a221d0a4eddb716f354bfab2f272701ec737065523fee49ac6f1d not found: ID does not exist" Oct 30 11:21:28.309502 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.309500 4289 scope.go:122] "RemoveContainer" containerID="c2704b6337a1d4056df4fd02608274c4e7b6f9745759466418469b66cd5aea53" Oct 30 11:21:28.309710 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.309695 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2704b6337a1d4056df4fd02608274c4e7b6f9745759466418469b66cd5aea53"} err="failed to get container status \"c2704b6337a1d4056df4fd02608274c4e7b6f9745759466418469b66cd5aea53\": rpc error: code = NotFound desc = could not find container \"c2704b6337a1d4056df4fd02608274c4e7b6f9745759466418469b66cd5aea53\": container with ID starting with c2704b6337a1d4056df4fd02608274c4e7b6f9745759466418469b66cd5aea53 not found: ID does not exist" Oct 30 11:21:28.309710 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.309708 4289 scope.go:122] "RemoveContainer" containerID="1791222d2e7cfe62c28cdc74f53766e585b66eb6553f736591ed8c0fbedbdeaf" Oct 30 11:21:28.309907 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.309886 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1791222d2e7cfe62c28cdc74f53766e585b66eb6553f736591ed8c0fbedbdeaf"} err="failed to get container status \"1791222d2e7cfe62c28cdc74f53766e585b66eb6553f736591ed8c0fbedbdeaf\": rpc error: code = NotFound desc = could not find container \"1791222d2e7cfe62c28cdc74f53766e585b66eb6553f736591ed8c0fbedbdeaf\": container with ID starting with 1791222d2e7cfe62c28cdc74f53766e585b66eb6553f736591ed8c0fbedbdeaf not found: ID does not exist" Oct 30 11:21:28.309978 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.309908 4289 scope.go:122] "RemoveContainer" containerID="9d06a0d05fa3800fd49de39549fd2cf3a5c4832756b4a5c264941154a1916e18" Oct 30 11:21:28.319745 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.319728 4289 scope.go:122] "RemoveContainer" containerID="9d06a0d05fa3800fd49de39549fd2cf3a5c4832756b4a5c264941154a1916e18" Oct 30 11:21:28.320016 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:21:28.319999 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d06a0d05fa3800fd49de39549fd2cf3a5c4832756b4a5c264941154a1916e18\": container with ID starting with 9d06a0d05fa3800fd49de39549fd2cf3a5c4832756b4a5c264941154a1916e18 not found: ID does not exist" containerID="9d06a0d05fa3800fd49de39549fd2cf3a5c4832756b4a5c264941154a1916e18" Oct 30 11:21:28.320057 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.320022 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d06a0d05fa3800fd49de39549fd2cf3a5c4832756b4a5c264941154a1916e18"} err="failed to get container status \"9d06a0d05fa3800fd49de39549fd2cf3a5c4832756b4a5c264941154a1916e18\": rpc error: code = NotFound desc = could not find container \"9d06a0d05fa3800fd49de39549fd2cf3a5c4832756b4a5c264941154a1916e18\": container with ID starting with 9d06a0d05fa3800fd49de39549fd2cf3a5c4832756b4a5c264941154a1916e18 not found: ID does not exist" Oct 30 11:21:28.320057 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.320037 4289 scope.go:122] "RemoveContainer" containerID="ebfb21866551ae4032c89f5f23f4ce8fa7054712d322e20faea5bda9b1fdb57e" Oct 30 11:21:28.326840 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.326771 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:28.327404 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.327366 4289 scope.go:122] "RemoveContainer" containerID="549abeaf141117579caf86f961c41e4a8e9ec84e775ac34957b32011d3de6651" Oct 30 11:21:28.334388 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.334373 4289 scope.go:122] "RemoveContainer" containerID="82ca5403ace38c16289edd766983e820a58322f53c55fbfef4211ba7f952c3a8" Oct 30 11:21:28.341723 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.341702 4289 scope.go:122] "RemoveContainer" containerID="ebfb21866551ae4032c89f5f23f4ce8fa7054712d322e20faea5bda9b1fdb57e" Oct 30 11:21:28.341979 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:21:28.341962 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebfb21866551ae4032c89f5f23f4ce8fa7054712d322e20faea5bda9b1fdb57e\": container with ID starting with ebfb21866551ae4032c89f5f23f4ce8fa7054712d322e20faea5bda9b1fdb57e not found: ID does not exist" containerID="ebfb21866551ae4032c89f5f23f4ce8fa7054712d322e20faea5bda9b1fdb57e" Oct 30 11:21:28.342066 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.341986 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebfb21866551ae4032c89f5f23f4ce8fa7054712d322e20faea5bda9b1fdb57e"} err="failed to get container status \"ebfb21866551ae4032c89f5f23f4ce8fa7054712d322e20faea5bda9b1fdb57e\": rpc error: code = NotFound desc = could not find container \"ebfb21866551ae4032c89f5f23f4ce8fa7054712d322e20faea5bda9b1fdb57e\": container with ID starting with ebfb21866551ae4032c89f5f23f4ce8fa7054712d322e20faea5bda9b1fdb57e not found: ID does not exist" Oct 30 11:21:28.342066 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.342003 4289 scope.go:122] "RemoveContainer" containerID="549abeaf141117579caf86f961c41e4a8e9ec84e775ac34957b32011d3de6651" Oct 30 11:21:28.342262 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:21:28.342241 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"549abeaf141117579caf86f961c41e4a8e9ec84e775ac34957b32011d3de6651\": container with ID starting with 549abeaf141117579caf86f961c41e4a8e9ec84e775ac34957b32011d3de6651 not found: ID does not exist" containerID="549abeaf141117579caf86f961c41e4a8e9ec84e775ac34957b32011d3de6651" Oct 30 11:21:28.342320 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.342267 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"549abeaf141117579caf86f961c41e4a8e9ec84e775ac34957b32011d3de6651"} err="failed to get container status \"549abeaf141117579caf86f961c41e4a8e9ec84e775ac34957b32011d3de6651\": rpc error: code = NotFound desc = could not find container \"549abeaf141117579caf86f961c41e4a8e9ec84e775ac34957b32011d3de6651\": container with ID starting with 549abeaf141117579caf86f961c41e4a8e9ec84e775ac34957b32011d3de6651 not found: ID does not exist" Oct 30 11:21:28.342320 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.342286 4289 scope.go:122] "RemoveContainer" containerID="82ca5403ace38c16289edd766983e820a58322f53c55fbfef4211ba7f952c3a8" Oct 30 11:21:28.342615 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:21:28.342595 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82ca5403ace38c16289edd766983e820a58322f53c55fbfef4211ba7f952c3a8\": container with ID starting with 82ca5403ace38c16289edd766983e820a58322f53c55fbfef4211ba7f952c3a8 not found: ID does not exist" containerID="82ca5403ace38c16289edd766983e820a58322f53c55fbfef4211ba7f952c3a8" Oct 30 11:21:28.342685 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.342621 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82ca5403ace38c16289edd766983e820a58322f53c55fbfef4211ba7f952c3a8"} err="failed to get container status \"82ca5403ace38c16289edd766983e820a58322f53c55fbfef4211ba7f952c3a8\": rpc error: code = NotFound desc = could not find container \"82ca5403ace38c16289edd766983e820a58322f53c55fbfef4211ba7f952c3a8\": container with ID starting with 82ca5403ace38c16289edd766983e820a58322f53c55fbfef4211ba7f952c3a8 not found: ID does not exist" Oct 30 11:21:28.342685 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.342638 4289 scope.go:122] "RemoveContainer" containerID="ebfb21866551ae4032c89f5f23f4ce8fa7054712d322e20faea5bda9b1fdb57e" Oct 30 11:21:28.342930 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.342910 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebfb21866551ae4032c89f5f23f4ce8fa7054712d322e20faea5bda9b1fdb57e"} err="failed to get container status \"ebfb21866551ae4032c89f5f23f4ce8fa7054712d322e20faea5bda9b1fdb57e\": rpc error: code = NotFound desc = could not find container \"ebfb21866551ae4032c89f5f23f4ce8fa7054712d322e20faea5bda9b1fdb57e\": container with ID starting with ebfb21866551ae4032c89f5f23f4ce8fa7054712d322e20faea5bda9b1fdb57e not found: ID does not exist" Oct 30 11:21:28.342930 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.342929 4289 scope.go:122] "RemoveContainer" containerID="549abeaf141117579caf86f961c41e4a8e9ec84e775ac34957b32011d3de6651" Oct 30 11:21:28.343184 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.343164 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"549abeaf141117579caf86f961c41e4a8e9ec84e775ac34957b32011d3de6651"} err="failed to get container status \"549abeaf141117579caf86f961c41e4a8e9ec84e775ac34957b32011d3de6651\": rpc error: code = NotFound desc = could not find container \"549abeaf141117579caf86f961c41e4a8e9ec84e775ac34957b32011d3de6651\": container with ID starting with 549abeaf141117579caf86f961c41e4a8e9ec84e775ac34957b32011d3de6651 not found: ID does not exist" Oct 30 11:21:28.343249 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.343183 4289 scope.go:122] "RemoveContainer" containerID="82ca5403ace38c16289edd766983e820a58322f53c55fbfef4211ba7f952c3a8" Oct 30 11:21:28.343459 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.343418 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82ca5403ace38c16289edd766983e820a58322f53c55fbfef4211ba7f952c3a8"} err="failed to get container status \"82ca5403ace38c16289edd766983e820a58322f53c55fbfef4211ba7f952c3a8\": rpc error: code = NotFound desc = could not find container \"82ca5403ace38c16289edd766983e820a58322f53c55fbfef4211ba7f952c3a8\": container with ID starting with 82ca5403ace38c16289edd766983e820a58322f53c55fbfef4211ba7f952c3a8 not found: ID does not exist" Oct 30 11:21:28.343518 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.343460 4289 scope.go:122] "RemoveContainer" containerID="ebfb21866551ae4032c89f5f23f4ce8fa7054712d322e20faea5bda9b1fdb57e" Oct 30 11:21:28.344246 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.344224 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebfb21866551ae4032c89f5f23f4ce8fa7054712d322e20faea5bda9b1fdb57e"} err="failed to get container status \"ebfb21866551ae4032c89f5f23f4ce8fa7054712d322e20faea5bda9b1fdb57e\": rpc error: code = NotFound desc = could not find container \"ebfb21866551ae4032c89f5f23f4ce8fa7054712d322e20faea5bda9b1fdb57e\": container with ID starting with ebfb21866551ae4032c89f5f23f4ce8fa7054712d322e20faea5bda9b1fdb57e not found: ID does not exist" Oct 30 11:21:28.344246 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.344245 4289 scope.go:122] "RemoveContainer" containerID="549abeaf141117579caf86f961c41e4a8e9ec84e775ac34957b32011d3de6651" Oct 30 11:21:28.344643 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.344616 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"549abeaf141117579caf86f961c41e4a8e9ec84e775ac34957b32011d3de6651"} err="failed to get container status \"549abeaf141117579caf86f961c41e4a8e9ec84e775ac34957b32011d3de6651\": rpc error: code = NotFound desc = could not find container \"549abeaf141117579caf86f961c41e4a8e9ec84e775ac34957b32011d3de6651\": container with ID starting with 549abeaf141117579caf86f961c41e4a8e9ec84e775ac34957b32011d3de6651 not found: ID does not exist" Oct 30 11:21:28.344739 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.344645 4289 scope.go:122] "RemoveContainer" containerID="82ca5403ace38c16289edd766983e820a58322f53c55fbfef4211ba7f952c3a8" Oct 30 11:21:28.344934 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.344904 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82ca5403ace38c16289edd766983e820a58322f53c55fbfef4211ba7f952c3a8"} err="failed to get container status \"82ca5403ace38c16289edd766983e820a58322f53c55fbfef4211ba7f952c3a8\": rpc error: code = NotFound desc = could not find container \"82ca5403ace38c16289edd766983e820a58322f53c55fbfef4211ba7f952c3a8\": container with ID starting with 82ca5403ace38c16289edd766983e820a58322f53c55fbfef4211ba7f952c3a8 not found: ID does not exist" Oct 30 11:21:28.350083 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.350063 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts"] Oct 30 11:21:28.360984 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.360961 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-2wfts"] Oct 30 11:21:28.396106 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.396073 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf"] Oct 30 11:21:28.411022 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.406005 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-zmxqf"] Oct 30 11:21:28.438250 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.438217 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd"] Oct 30 11:21:28.444860 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.444831 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-bh5wd"] Oct 30 11:21:28.479125 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.479095 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx"] Oct 30 11:21:28.481753 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.481726 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-lcljx"] Oct 30 11:21:28.490776 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.490745 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:28.557217 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.557164 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:21:28.660472 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.660410 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:28.707966 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.707940 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17bb13d0-fa33-4e16-bf43-8ca7189291b8-config-out\" (UniqueName: \"kubernetes.io/empty-dir/17bb13d0-fa33-4e16-bf43-8ca7189291b8-config-out\") pod \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " Oct 30 11:21:28.708115 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.707979 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-thanos-prometheus-http-client-file\") pod \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " Oct 30 11:21:28.708115 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.707997 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17bb13d0-fa33-4e16-bf43-8ca7189291b8-tls-assets\" (UniqueName: \"kubernetes.io/projected/17bb13d0-fa33-4e16-bf43-8ca7189291b8-tls-assets\") pod \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " Oct 30 11:21:28.708115 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.708051 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " Oct 30 11:21:28.708115 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.708081 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-secret-prometheus-web-tls-ca\") pod \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " Oct 30 11:21:28.708115 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.708111 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17bb13d0-fa33-4e16-bf43-8ca7189291b8-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/17bb13d0-fa33-4e16-bf43-8ca7189291b8-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " Oct 30 11:21:28.708875 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.708148 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " Oct 30 11:21:28.708875 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.708212 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17bb13d0-fa33-4e16-bf43-8ca7189291b8-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/17bb13d0-fa33-4e16-bf43-8ca7189291b8-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " Oct 30 11:21:28.708875 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.708243 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-config\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-config\") pod \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " Oct 30 11:21:28.708875 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.708276 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-web-config\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-web-config\") pod \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " Oct 30 11:21:28.708875 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.708315 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17bb13d0-fa33-4e16-bf43-8ca7189291b8-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/17bb13d0-fa33-4e16-bf43-8ca7189291b8-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " Oct 30 11:21:28.708875 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.708369 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17bb13d0-fa33-4e16-bf43-8ca7189291b8-kube-api-access-9tv5r\" (UniqueName: \"kubernetes.io/projected/17bb13d0-fa33-4e16-bf43-8ca7189291b8-kube-api-access-9tv5r\") pod \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " Oct 30 11:21:28.708875 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.708480 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396\") pod \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\" (UID: \"17bb13d0-fa33-4e16-bf43-8ca7189291b8\") " Oct 30 11:21:28.708875 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.708574 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17bb13d0-fa33-4e16-bf43-8ca7189291b8-prometheus-data-science-monitoringstack-rulefiles-2" pod "17bb13d0-fa33-4e16-bf43-8ca7189291b8" (UID: "17bb13d0-fa33-4e16-bf43-8ca7189291b8"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:21:28.708875 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.708865 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/17bb13d0-fa33-4e16-bf43-8ca7189291b8-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:28.710145 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.710119 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17bb13d0-fa33-4e16-bf43-8ca7189291b8-prometheus-data-science-monitoringstack-rulefiles-0" pod "17bb13d0-fa33-4e16-bf43-8ca7189291b8" (UID: "17bb13d0-fa33-4e16-bf43-8ca7189291b8"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:21:28.710460 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.710412 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17bb13d0-fa33-4e16-bf43-8ca7189291b8-prometheus-data-science-monitoringstack-rulefiles-1" pod "17bb13d0-fa33-4e16-bf43-8ca7189291b8" (UID: "17bb13d0-fa33-4e16-bf43-8ca7189291b8"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:21:28.710846 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.710823 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17bb13d0-fa33-4e16-bf43-8ca7189291b8-config-out" pod "17bb13d0-fa33-4e16-bf43-8ca7189291b8" (UID: "17bb13d0-fa33-4e16-bf43-8ca7189291b8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:21:28.711041 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.711013 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "17bb13d0-fa33-4e16-bf43-8ca7189291b8" (UID: "17bb13d0-fa33-4e16-bf43-8ca7189291b8"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:21:28.711199 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.711174 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-thanos-prometheus-http-client-file" pod "17bb13d0-fa33-4e16-bf43-8ca7189291b8" (UID: "17bb13d0-fa33-4e16-bf43-8ca7189291b8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:21:28.712033 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.712008 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17bb13d0-fa33-4e16-bf43-8ca7189291b8-tls-assets" pod "17bb13d0-fa33-4e16-bf43-8ca7189291b8" (UID: "17bb13d0-fa33-4e16-bf43-8ca7189291b8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:21:28.712033 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.712019 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-secret-prometheus-web-tls-ca" pod "17bb13d0-fa33-4e16-bf43-8ca7189291b8" (UID: "17bb13d0-fa33-4e16-bf43-8ca7189291b8"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:21:28.712542 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.712522 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-config" pod "17bb13d0-fa33-4e16-bf43-8ca7189291b8" (UID: "17bb13d0-fa33-4e16-bf43-8ca7189291b8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:21:28.712793 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.712774 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17bb13d0-fa33-4e16-bf43-8ca7189291b8-kube-api-access-9tv5r" pod "17bb13d0-fa33-4e16-bf43-8ca7189291b8" (UID: "17bb13d0-fa33-4e16-bf43-8ca7189291b8"). InnerVolumeSpecName "kube-api-access-9tv5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:21:28.713176 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.713148 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "17bb13d0-fa33-4e16-bf43-8ca7189291b8" (UID: "17bb13d0-fa33-4e16-bf43-8ca7189291b8"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:21:28.714104 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.714082 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396" pod "17bb13d0-fa33-4e16-bf43-8ca7189291b8" (UID: "17bb13d0-fa33-4e16-bf43-8ca7189291b8"). InnerVolumeSpecName "pvc-ae2fc6cc-1a29-41df-b109-2b8489d4e098". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 11:21:28.723901 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.723881 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-web-config" pod "17bb13d0-fa33-4e16-bf43-8ca7189291b8" (UID: "17bb13d0-fa33-4e16-bf43-8ca7189291b8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:21:28.809538 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.809454 4289 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-secret-prometheus-web-tls-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:28.809538 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.809486 4289 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:28.809538 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.809502 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/17bb13d0-fa33-4e16-bf43-8ca7189291b8-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:28.809538 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.809514 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:28.809538 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.809524 4289 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-web-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:28.809538 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.809534 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/17bb13d0-fa33-4e16-bf43-8ca7189291b8-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:28.809538 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.809543 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tv5r\" (UniqueName: \"kubernetes.io/projected/17bb13d0-fa33-4e16-bf43-8ca7189291b8-kube-api-access-9tv5r\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:28.809848 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.809579 4289 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-ae2fc6cc-1a29-41df-b109-2b8489d4e098\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396\") on node \"ip-10-0-137-242.ec2.internal\" " Oct 30 11:21:28.809848 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.809591 4289 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17bb13d0-fa33-4e16-bf43-8ca7189291b8-config-out\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:28.809848 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.809600 4289 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:28.809848 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.809609 4289 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17bb13d0-fa33-4e16-bf43-8ca7189291b8-tls-assets\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:28.809848 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.809619 4289 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/17bb13d0-fa33-4e16-bf43-8ca7189291b8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:28.820692 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.820675 4289 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-ae2fc6cc-1a29-41df-b109-2b8489d4e098" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396") on node "ip-10-0-137-242.ec2.internal" Oct 30 11:21:28.910471 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:28.910420 4289 reconciler_common.go:299] "Volume detached for volume \"pvc-ae2fc6cc-1a29-41df-b109-2b8489d4e098\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"csi-41fb3b285706a9985249f9bffb659d03160e9e3e6288d3728f95a76a8535f583\"" Oct 30 11:21:29.277817 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:29.277781 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"17bb13d0-fa33-4e16-bf43-8ca7189291b8","Type":"ContainerDied","Data":"8b7e568319d555f78071f4c169eeed3cf77bc124966f04aa30e826a01073cc2a"} Oct 30 11:21:29.277979 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:29.277863 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:21:29.277979 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:29.277873 4289 scope.go:122] "RemoveContainer" containerID="4574094aa55387e332d1ab1293e4e9a2bd93519893f9854c6d87b3305e809f96" Oct 30 11:21:29.286725 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:29.286700 4289 scope.go:122] "RemoveContainer" containerID="bf9e931a2ea2474f6e1100e2eac395dc2801a4d76d4b96f407ba9494afcbbe93" Oct 30 11:21:29.298210 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:29.298074 4289 scope.go:122] "RemoveContainer" containerID="b7ca7ed72ca49de380b8bde5ab25b98294177e6996e90c76afca8e25191302ba" Oct 30 11:21:29.308218 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:29.308195 4289 scope.go:122] "RemoveContainer" containerID="731d3b1901d6a398028250fe34008e65503edc76e1656158da8093ae3448b024" Oct 30 11:21:29.332188 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:29.332165 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 11:21:29.338239 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:29.338212 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 11:21:29.362313 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:29.362278 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:29.563580 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:29.563499 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:29.629326 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:29.629295 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17bb13d0-fa33-4e16-bf43-8ca7189291b8" path="/var/lib/kubelet/pods/17bb13d0-fa33-4e16-bf43-8ca7189291b8/volumes" Oct 30 11:21:29.629841 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:29.629826 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b34e08e-81fb-4d17-8f15-2e24e94f47c0" path="/var/lib/kubelet/pods/1b34e08e-81fb-4d17-8f15-2e24e94f47c0/volumes" Oct 30 11:21:29.630466 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:29.630449 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22a70032-0628-4014-8cf5-aaac0fe64900" path="/var/lib/kubelet/pods/22a70032-0628-4014-8cf5-aaac0fe64900/volumes" Oct 30 11:21:29.630814 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:29.630802 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a33defc9-4a83-4e81-9095-aa32e383792c" path="/var/lib/kubelet/pods/a33defc9-4a83-4e81-9095-aa32e383792c/volumes" Oct 30 11:21:29.631207 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:29.631196 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e962d8b6-4549-4ec3-8e7e-bf003453f7a0" path="/var/lib/kubelet/pods/e962d8b6-4549-4ec3-8e7e-bf003453f7a0/volumes" Oct 30 11:21:29.981329 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:29.981291 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.86:9093/-/ready\": dial tcp 10.134.0.86:9093: connect: connection refused" Oct 30 11:21:30.227933 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:30.227894 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:34.981310 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:34.981271 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.86:9093/-/ready\": dial tcp 10.134.0.86:9093: connect: connection refused" Oct 30 11:21:38.014088 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.014052 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:21:38.189110 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.189031 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-cluster-tls-config\") pod \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " Oct 30 11:21:38.189110 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.189086 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-alertmanager-data-science-monitoringstack-db\") pod \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " Oct 30 11:21:38.189310 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.189118 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-config-out\" (UniqueName: \"kubernetes.io/empty-dir/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-config-out\") pod \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " Oct 30 11:21:38.189310 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.189162 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-kube-api-access-whx6j\" (UniqueName: \"kubernetes.io/projected/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-kube-api-access-whx6j\") pod \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " Oct 30 11:21:38.189310 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.189226 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-config-volume\" (UniqueName: \"kubernetes.io/secret/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-config-volume\") pod \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " Oct 30 11:21:38.189310 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.189255 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-tls-assets\" (UniqueName: \"kubernetes.io/projected/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-tls-assets\") pod \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " Oct 30 11:21:38.189310 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.189286 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-web-config\" (UniqueName: \"kubernetes.io/secret/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-web-config\") pod \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\" (UID: \"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e\") " Oct 30 11:21:38.189642 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.189529 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-alertmanager-data-science-monitoringstack-db" pod "d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e" (UID: "d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:21:38.191457 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.191409 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-kube-api-access-whx6j" pod "d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e" (UID: "d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e"). InnerVolumeSpecName "kube-api-access-whx6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:21:38.191773 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.191731 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-config-out" pod "d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e" (UID: "d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:21:38.191773 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.191743 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-tls-assets" pod "d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e" (UID: "d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:21:38.191773 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.191751 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-config-volume" pod "d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e" (UID: "d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:21:38.195903 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.195880 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-cluster-tls-config" pod "d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e" (UID: "d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:21:38.201979 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.201957 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-web-config" pod "d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e" (UID: "d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:21:38.290275 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.290225 4289 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:38.290275 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.290269 4289 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-config-out\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:38.290275 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.290285 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whx6j\" (UniqueName: \"kubernetes.io/projected/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-kube-api-access-whx6j\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:38.290545 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.290298 4289 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:38.290545 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.290309 4289 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-tls-assets\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:38.290545 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.290344 4289 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-web-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:38.290545 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.290357 4289 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e-cluster-tls-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:21:38.323929 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.323899 4289 generic.go:356] "Generic (PLEG): container finished" podID="d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e" containerID="fffe9a0182189288cd2d58b2ec8df231662eafabfc285faba9de6bce7501fc8a" exitCode=0 Oct 30 11:21:38.324050 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.323965 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:21:38.324050 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.323988 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e","Type":"ContainerDied","Data":"fffe9a0182189288cd2d58b2ec8df231662eafabfc285faba9de6bce7501fc8a"} Oct 30 11:21:38.324050 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.324041 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e","Type":"ContainerDied","Data":"3c2d81592a23d7b4bc133659dec176ad9b78931b118caca7e15e101ff4a24fff"} Oct 30 11:21:38.324163 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.324066 4289 scope.go:122] "RemoveContainer" containerID="939084d548401ebe7f0f0c46fb8e897145291e00d948f86ff961c44c2c88c400" Oct 30 11:21:38.332930 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.332851 4289 scope.go:122] "RemoveContainer" containerID="fffe9a0182189288cd2d58b2ec8df231662eafabfc285faba9de6bce7501fc8a" Oct 30 11:21:38.340602 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.340583 4289 scope.go:122] "RemoveContainer" containerID="d329c4449e57e94feaec69a0a7582ae3db7d3c035ad09cc8317d017219c36532" Oct 30 11:21:38.347624 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.347602 4289 scope.go:122] "RemoveContainer" containerID="939084d548401ebe7f0f0c46fb8e897145291e00d948f86ff961c44c2c88c400" Oct 30 11:21:38.347902 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:21:38.347883 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"939084d548401ebe7f0f0c46fb8e897145291e00d948f86ff961c44c2c88c400\": container with ID starting with 939084d548401ebe7f0f0c46fb8e897145291e00d948f86ff961c44c2c88c400 not found: ID does not exist" containerID="939084d548401ebe7f0f0c46fb8e897145291e00d948f86ff961c44c2c88c400" Oct 30 11:21:38.347953 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.347911 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"939084d548401ebe7f0f0c46fb8e897145291e00d948f86ff961c44c2c88c400"} err="failed to get container status \"939084d548401ebe7f0f0c46fb8e897145291e00d948f86ff961c44c2c88c400\": rpc error: code = NotFound desc = could not find container \"939084d548401ebe7f0f0c46fb8e897145291e00d948f86ff961c44c2c88c400\": container with ID starting with 939084d548401ebe7f0f0c46fb8e897145291e00d948f86ff961c44c2c88c400 not found: ID does not exist" Oct 30 11:21:38.347953 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.347928 4289 scope.go:122] "RemoveContainer" containerID="fffe9a0182189288cd2d58b2ec8df231662eafabfc285faba9de6bce7501fc8a" Oct 30 11:21:38.348178 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:21:38.348158 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fffe9a0182189288cd2d58b2ec8df231662eafabfc285faba9de6bce7501fc8a\": container with ID starting with fffe9a0182189288cd2d58b2ec8df231662eafabfc285faba9de6bce7501fc8a not found: ID does not exist" containerID="fffe9a0182189288cd2d58b2ec8df231662eafabfc285faba9de6bce7501fc8a" Oct 30 11:21:38.348219 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.348185 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fffe9a0182189288cd2d58b2ec8df231662eafabfc285faba9de6bce7501fc8a"} err="failed to get container status \"fffe9a0182189288cd2d58b2ec8df231662eafabfc285faba9de6bce7501fc8a\": rpc error: code = NotFound desc = could not find container \"fffe9a0182189288cd2d58b2ec8df231662eafabfc285faba9de6bce7501fc8a\": container with ID starting with fffe9a0182189288cd2d58b2ec8df231662eafabfc285faba9de6bce7501fc8a not found: ID does not exist" Oct 30 11:21:38.348219 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.348200 4289 scope.go:122] "RemoveContainer" containerID="d329c4449e57e94feaec69a0a7582ae3db7d3c035ad09cc8317d017219c36532" Oct 30 11:21:38.348477 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:21:38.348461 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d329c4449e57e94feaec69a0a7582ae3db7d3c035ad09cc8317d017219c36532\": container with ID starting with d329c4449e57e94feaec69a0a7582ae3db7d3c035ad09cc8317d017219c36532 not found: ID does not exist" containerID="d329c4449e57e94feaec69a0a7582ae3db7d3c035ad09cc8317d017219c36532" Oct 30 11:21:38.348540 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.348481 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d329c4449e57e94feaec69a0a7582ae3db7d3c035ad09cc8317d017219c36532"} err="failed to get container status \"d329c4449e57e94feaec69a0a7582ae3db7d3c035ad09cc8317d017219c36532\": rpc error: code = NotFound desc = could not find container \"d329c4449e57e94feaec69a0a7582ae3db7d3c035ad09cc8317d017219c36532\": container with ID starting with d329c4449e57e94feaec69a0a7582ae3db7d3c035ad09cc8317d017219c36532 not found: ID does not exist" Oct 30 11:21:38.374288 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.374268 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:21:38.386177 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:38.386158 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:21:39.223451 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:39.223403 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:39.624064 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:39.623987 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e" path="/var/lib/kubelet/pods/d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e/volumes" Oct 30 11:21:51.103983 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.103940 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x"] Oct 30 11:21:51.104465 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104393 4289 cpu_manager.go:414] "containerMap: removing container" podUID="e962d8b6-4549-4ec3-8e7e-bf003453f7a0" containerName="method-gate" Oct 30 11:21:51.104465 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104410 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e962d8b6-4549-4ec3-8e7e-bf003453f7a0" containerName="method-gate" Oct 30 11:21:51.104465 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104421 4289 cpu_manager.go:414] "containerMap: removing container" podUID="17bb13d0-fa33-4e16-bf43-8ca7189291b8" containerName="config-reloader" Oct 30 11:21:51.104465 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104454 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17bb13d0-fa33-4e16-bf43-8ca7189291b8" containerName="config-reloader" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104475 4289 cpu_manager.go:414] "containerMap: removing container" podUID="e962d8b6-4549-4ec3-8e7e-bf003453f7a0" containerName="kube-rbac-proxy" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104482 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e962d8b6-4549-4ec3-8e7e-bf003453f7a0" containerName="kube-rbac-proxy" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104492 4289 cpu_manager.go:414] "containerMap: removing container" podUID="a33defc9-4a83-4e81-9095-aa32e383792c" containerName="method-gate" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104500 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a33defc9-4a83-4e81-9095-aa32e383792c" containerName="method-gate" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104509 4289 cpu_manager.go:414] "containerMap: removing container" podUID="1b34e08e-81fb-4d17-8f15-2e24e94f47c0" containerName="korrel8r" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104515 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b34e08e-81fb-4d17-8f15-2e24e94f47c0" containerName="korrel8r" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104532 4289 cpu_manager.go:414] "containerMap: removing container" podUID="d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e" containerName="init-config-reloader" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104539 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e" containerName="init-config-reloader" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104549 4289 cpu_manager.go:414] "containerMap: removing container" podUID="d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e" containerName="config-reloader" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104555 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e" containerName="config-reloader" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104564 4289 cpu_manager.go:414] "containerMap: removing container" podUID="17bb13d0-fa33-4e16-bf43-8ca7189291b8" containerName="prometheus" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104572 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17bb13d0-fa33-4e16-bf43-8ca7189291b8" containerName="prometheus" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104585 4289 cpu_manager.go:414] "containerMap: removing container" podUID="a33defc9-4a83-4e81-9095-aa32e383792c" containerName="prom-label-proxy" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104591 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a33defc9-4a83-4e81-9095-aa32e383792c" containerName="prom-label-proxy" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104602 4289 cpu_manager.go:414] "containerMap: removing container" podUID="22a70032-0628-4014-8cf5-aaac0fe64900" containerName="kube-rbac-proxy" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104611 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a70032-0628-4014-8cf5-aaac0fe64900" containerName="kube-rbac-proxy" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104628 4289 cpu_manager.go:414] "containerMap: removing container" podUID="a33defc9-4a83-4e81-9095-aa32e383792c" containerName="kube-rbac-proxy" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104636 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a33defc9-4a83-4e81-9095-aa32e383792c" containerName="kube-rbac-proxy" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104645 4289 cpu_manager.go:414] "containerMap: removing container" podUID="d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e" containerName="alertmanager" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104652 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e" containerName="alertmanager" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104680 4289 cpu_manager.go:414] "containerMap: removing container" podUID="17bb13d0-fa33-4e16-bf43-8ca7189291b8" containerName="init-config-reloader" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104688 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17bb13d0-fa33-4e16-bf43-8ca7189291b8" containerName="init-config-reloader" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104698 4289 cpu_manager.go:414] "containerMap: removing container" podUID="17bb13d0-fa33-4e16-bf43-8ca7189291b8" containerName="thanos-sidecar" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104705 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17bb13d0-fa33-4e16-bf43-8ca7189291b8" containerName="thanos-sidecar" Oct 30 11:21:51.104708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104715 4289 cpu_manager.go:414] "containerMap: removing container" podUID="e962d8b6-4549-4ec3-8e7e-bf003453f7a0" containerName="prom-label-proxy" Oct 30 11:21:51.105856 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104722 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e962d8b6-4549-4ec3-8e7e-bf003453f7a0" containerName="prom-label-proxy" Oct 30 11:21:51.105856 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104812 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="17bb13d0-fa33-4e16-bf43-8ca7189291b8" containerName="thanos-sidecar" Oct 30 11:21:51.105856 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104823 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="e962d8b6-4549-4ec3-8e7e-bf003453f7a0" containerName="prom-label-proxy" Oct 30 11:21:51.105856 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104835 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="e962d8b6-4549-4ec3-8e7e-bf003453f7a0" containerName="method-gate" Oct 30 11:21:51.105856 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104843 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="17bb13d0-fa33-4e16-bf43-8ca7189291b8" containerName="config-reloader" Oct 30 11:21:51.105856 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104854 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="e962d8b6-4549-4ec3-8e7e-bf003453f7a0" containerName="kube-rbac-proxy" Oct 30 11:21:51.105856 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104861 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="a33defc9-4a83-4e81-9095-aa32e383792c" containerName="method-gate" Oct 30 11:21:51.105856 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104868 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b34e08e-81fb-4d17-8f15-2e24e94f47c0" containerName="korrel8r" Oct 30 11:21:51.105856 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104876 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e" containerName="config-reloader" Oct 30 11:21:51.105856 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104884 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="17bb13d0-fa33-4e16-bf43-8ca7189291b8" containerName="prometheus" Oct 30 11:21:51.105856 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104892 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="a33defc9-4a83-4e81-9095-aa32e383792c" containerName="prom-label-proxy" Oct 30 11:21:51.105856 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104899 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a70032-0628-4014-8cf5-aaac0fe64900" containerName="kube-rbac-proxy" Oct 30 11:21:51.105856 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104908 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="a33defc9-4a83-4e81-9095-aa32e383792c" containerName="kube-rbac-proxy" Oct 30 11:21:51.105856 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.104916 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="d67f9c21-0c37-4c52-92c3-6c31c9dc2d2e" containerName="alertmanager" Oct 30 11:21:51.109784 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.109759 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" Oct 30 11:21:51.139489 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.139460 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x"] Oct 30 11:21:51.173043 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.173012 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:51.189166 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.189139 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/a517cbd2-329c-4c01-8f6f-955220592158-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-jtn4x\" (UID: \"a517cbd2-329c-4c01-8f6f-955220592158\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" Oct 30 11:21:51.189255 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.189171 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a517cbd2-329c-4c01-8f6f-955220592158-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-jtn4x\" (UID: \"a517cbd2-329c-4c01-8f6f-955220592158\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" Oct 30 11:21:51.189302 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.189258 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/a517cbd2-329c-4c01-8f6f-955220592158-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-jtn4x\" (UID: \"a517cbd2-329c-4c01-8f6f-955220592158\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" Oct 30 11:21:51.189335 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.189307 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wzd5\" (UniqueName: \"kubernetes.io/projected/a517cbd2-329c-4c01-8f6f-955220592158-kube-api-access-2wzd5\") pod \"data-science-lokistack-distributor-58947679b7-jtn4x\" (UID: \"a517cbd2-329c-4c01-8f6f-955220592158\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" Oct 30 11:21:51.189371 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.189335 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a517cbd2-329c-4c01-8f6f-955220592158-config\") pod \"data-science-lokistack-distributor-58947679b7-jtn4x\" (UID: \"a517cbd2-329c-4c01-8f6f-955220592158\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" Oct 30 11:21:51.202893 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.202871 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-distributor-grpc\"" Oct 30 11:21:51.202893 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.202884 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-distributor-http\"" Oct 30 11:21:51.203011 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.202897 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-dockercfg-4pmvt\"" Oct 30 11:21:51.205646 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.205627 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ca-bundle\"" Oct 30 11:21:51.205786 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.205769 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 11:21:51.206035 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.206020 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 11:21:51.214626 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.214609 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-config\"" Oct 30 11:21:51.289959 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.289929 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/a517cbd2-329c-4c01-8f6f-955220592158-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-jtn4x\" (UID: \"a517cbd2-329c-4c01-8f6f-955220592158\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" Oct 30 11:21:51.290117 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.289975 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wzd5\" (UniqueName: \"kubernetes.io/projected/a517cbd2-329c-4c01-8f6f-955220592158-kube-api-access-2wzd5\") pod \"data-science-lokistack-distributor-58947679b7-jtn4x\" (UID: \"a517cbd2-329c-4c01-8f6f-955220592158\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" Oct 30 11:21:51.290117 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.289998 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a517cbd2-329c-4c01-8f6f-955220592158-config\") pod \"data-science-lokistack-distributor-58947679b7-jtn4x\" (UID: \"a517cbd2-329c-4c01-8f6f-955220592158\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" Oct 30 11:21:51.290117 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.290022 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/a517cbd2-329c-4c01-8f6f-955220592158-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-jtn4x\" (UID: \"a517cbd2-329c-4c01-8f6f-955220592158\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" Oct 30 11:21:51.290117 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.290043 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a517cbd2-329c-4c01-8f6f-955220592158-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-jtn4x\" (UID: \"a517cbd2-329c-4c01-8f6f-955220592158\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" Oct 30 11:21:51.290882 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.290847 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a517cbd2-329c-4c01-8f6f-955220592158-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-jtn4x\" (UID: \"a517cbd2-329c-4c01-8f6f-955220592158\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" Oct 30 11:21:51.291019 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.290920 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a517cbd2-329c-4c01-8f6f-955220592158-config\") pod \"data-science-lokistack-distributor-58947679b7-jtn4x\" (UID: \"a517cbd2-329c-4c01-8f6f-955220592158\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" Oct 30 11:21:51.292536 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.292510 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/a517cbd2-329c-4c01-8f6f-955220592158-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-jtn4x\" (UID: \"a517cbd2-329c-4c01-8f6f-955220592158\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" Oct 30 11:21:51.292630 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.292588 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/a517cbd2-329c-4c01-8f6f-955220592158-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-jtn4x\" (UID: \"a517cbd2-329c-4c01-8f6f-955220592158\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" Oct 30 11:21:51.316502 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.316473 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wzd5\" (UniqueName: \"kubernetes.io/projected/a517cbd2-329c-4c01-8f6f-955220592158-kube-api-access-2wzd5\") pod \"data-science-lokistack-distributor-58947679b7-jtn4x\" (UID: \"a517cbd2-329c-4c01-8f6f-955220592158\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" Oct 30 11:21:51.383963 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.383881 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g"] Oct 30 11:21:51.388457 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.388417 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" Oct 30 11:21:51.391062 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.391031 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/2875bb2d-9cc3-4467-874a-bbacc4c94177-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-84985db556-4vt4g\" (UID: \"2875bb2d-9cc3-4467-874a-bbacc4c94177\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" Oct 30 11:21:51.391223 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.391075 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/2875bb2d-9cc3-4467-874a-bbacc4c94177-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-84985db556-4vt4g\" (UID: \"2875bb2d-9cc3-4467-874a-bbacc4c94177\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" Oct 30 11:21:51.391223 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.391092 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2875bb2d-9cc3-4467-874a-bbacc4c94177-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-84985db556-4vt4g\" (UID: \"2875bb2d-9cc3-4467-874a-bbacc4c94177\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" Oct 30 11:21:51.391321 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.391216 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t9ts\" (UniqueName: \"kubernetes.io/projected/2875bb2d-9cc3-4467-874a-bbacc4c94177-kube-api-access-2t9ts\") pod \"data-science-lokistack-querier-84985db556-4vt4g\" (UID: \"2875bb2d-9cc3-4467-874a-bbacc4c94177\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" Oct 30 11:21:51.391321 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.391255 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2875bb2d-9cc3-4467-874a-bbacc4c94177-config\") pod \"data-science-lokistack-querier-84985db556-4vt4g\" (UID: \"2875bb2d-9cc3-4467-874a-bbacc4c94177\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" Oct 30 11:21:51.391402 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.391378 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/2875bb2d-9cc3-4467-874a-bbacc4c94177-test-loki-shared-secret\") pod \"data-science-lokistack-querier-84985db556-4vt4g\" (UID: \"2875bb2d-9cc3-4467-874a-bbacc4c94177\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" Oct 30 11:21:51.396242 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.396205 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:51.411377 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.411350 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g"] Oct 30 11:21:51.421725 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.421693 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" Oct 30 11:21:51.482855 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.482833 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-shared-secret\"" Oct 30 11:21:51.483030 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.483015 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-querier-http\"" Oct 30 11:21:51.483111 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.483091 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-querier-grpc\"" Oct 30 11:21:51.495460 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.492785 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/2875bb2d-9cc3-4467-874a-bbacc4c94177-test-loki-shared-secret\") pod \"data-science-lokistack-querier-84985db556-4vt4g\" (UID: \"2875bb2d-9cc3-4467-874a-bbacc4c94177\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" Oct 30 11:21:51.495460 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.492859 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/2875bb2d-9cc3-4467-874a-bbacc4c94177-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-84985db556-4vt4g\" (UID: \"2875bb2d-9cc3-4467-874a-bbacc4c94177\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" Oct 30 11:21:51.495460 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.492905 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/2875bb2d-9cc3-4467-874a-bbacc4c94177-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-84985db556-4vt4g\" (UID: \"2875bb2d-9cc3-4467-874a-bbacc4c94177\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" Oct 30 11:21:51.495460 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.492932 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2875bb2d-9cc3-4467-874a-bbacc4c94177-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-84985db556-4vt4g\" (UID: \"2875bb2d-9cc3-4467-874a-bbacc4c94177\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" Oct 30 11:21:51.495460 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.492959 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t9ts\" (UniqueName: \"kubernetes.io/projected/2875bb2d-9cc3-4467-874a-bbacc4c94177-kube-api-access-2t9ts\") pod \"data-science-lokistack-querier-84985db556-4vt4g\" (UID: \"2875bb2d-9cc3-4467-874a-bbacc4c94177\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" Oct 30 11:21:51.495460 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.492985 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2875bb2d-9cc3-4467-874a-bbacc4c94177-config\") pod \"data-science-lokistack-querier-84985db556-4vt4g\" (UID: \"2875bb2d-9cc3-4467-874a-bbacc4c94177\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" Oct 30 11:21:51.495460 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.493849 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2875bb2d-9cc3-4467-874a-bbacc4c94177-config\") pod \"data-science-lokistack-querier-84985db556-4vt4g\" (UID: \"2875bb2d-9cc3-4467-874a-bbacc4c94177\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" Oct 30 11:21:51.495460 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.494411 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2875bb2d-9cc3-4467-874a-bbacc4c94177-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-84985db556-4vt4g\" (UID: \"2875bb2d-9cc3-4467-874a-bbacc4c94177\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" Oct 30 11:21:51.498937 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.498913 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/2875bb2d-9cc3-4467-874a-bbacc4c94177-test-loki-shared-secret\") pod \"data-science-lokistack-querier-84985db556-4vt4g\" (UID: \"2875bb2d-9cc3-4467-874a-bbacc4c94177\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" Oct 30 11:21:51.500318 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.500103 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/2875bb2d-9cc3-4467-874a-bbacc4c94177-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-84985db556-4vt4g\" (UID: \"2875bb2d-9cc3-4467-874a-bbacc4c94177\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" Oct 30 11:21:51.501699 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.501677 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/2875bb2d-9cc3-4467-874a-bbacc4c94177-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-84985db556-4vt4g\" (UID: \"2875bb2d-9cc3-4467-874a-bbacc4c94177\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" Oct 30 11:21:51.517215 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.517188 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t9ts\" (UniqueName: \"kubernetes.io/projected/2875bb2d-9cc3-4467-874a-bbacc4c94177-kube-api-access-2t9ts\") pod \"data-science-lokistack-querier-84985db556-4vt4g\" (UID: \"2875bb2d-9cc3-4467-874a-bbacc4c94177\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" Oct 30 11:21:51.575688 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.575657 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:51.701174 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.701098 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" Oct 30 11:21:51.729093 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.729058 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x"] Oct 30 11:21:51.729439 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:21:51.729403 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda517cbd2_329c_4c01_8f6f_955220592158.slice/crio-ce2b3dbc00322ae4e9ab5d16435dffeb13f020e2c417a8234b69250b07ac75c5 WatchSource:0}: Error finding container ce2b3dbc00322ae4e9ab5d16435dffeb13f020e2c417a8234b69250b07ac75c5: Status 404 returned error can't find the container with id ce2b3dbc00322ae4e9ab5d16435dffeb13f020e2c417a8234b69250b07ac75c5 Oct 30 11:21:51.731765 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.731748 4289 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 11:21:51.744751 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.744717 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj"] Oct 30 11:21:51.749336 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.749313 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:51.765859 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.765832 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\"" Oct 30 11:21:51.766015 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.765868 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-ca-bundle\"" Oct 30 11:21:51.766880 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.765897 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-http\"" Oct 30 11:21:51.766880 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.765937 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\"" Oct 30 11:21:51.767033 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.766019 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-client-http\"" Oct 30 11:21:51.767033 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.766120 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-dockercfg-5r9gx\"" Oct 30 11:21:51.767033 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.766938 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:51.770927 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.770902 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj"] Oct 30 11:21:51.795974 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.795941 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-rbac\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:51.796131 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.795980 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/551b0f59-4902-49a5-a83f-85bad32dee9e-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:51.796131 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.796086 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-tls-secret\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:51.796258 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.796164 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-lokistack-gateway\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:51.796258 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.796202 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss76v\" (UniqueName: \"kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-kube-api-access-ss76v\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:51.796369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.796308 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/551b0f59-4902-49a5-a83f-85bad32dee9e-tenants\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:51.796369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.796350 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:51.796504 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.796387 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:51.897180 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.897145 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss76v\" (UniqueName: \"kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-kube-api-access-ss76v\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:51.897383 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.897220 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/551b0f59-4902-49a5-a83f-85bad32dee9e-tenants\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:51.897383 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.897259 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:51.897383 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.897286 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:51.897383 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.897318 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-rbac\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:51.897383 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.897353 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/551b0f59-4902-49a5-a83f-85bad32dee9e-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:51.897383 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:21:51.897363 4289 projected.go:291] Couldn't get configMap redhat-ods-monitoring/data-science-lokistack-gateway-ca-bundle: configmap "data-science-lokistack-gateway-ca-bundle" not found Oct 30 11:21:51.897383 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:21:51.897384 4289 projected.go:196] Error preparing data for projected volume data-science-lokistack-gateway-ca-bundle for pod redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj: configmap "data-science-lokistack-gateway-ca-bundle" not found Oct 30 11:21:51.897847 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.897401 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-tls-secret\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:51.897847 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:21:51.897475 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-data-science-lokistack-gateway-ca-bundle podName:551b0f59-4902-49a5-a83f-85bad32dee9e nodeName:}" failed. No retries permitted until 2026-10-30 11:21:52.397454025 +0000 UTC m=+2202.898833508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "data-science-lokistack-gateway-ca-bundle" (UniqueName: "kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-data-science-lokistack-gateway-ca-bundle") pod "data-science-lokistack-gateway-698f8cbf65-btsfj" (UID: "551b0f59-4902-49a5-a83f-85bad32dee9e") : configmap "data-science-lokistack-gateway-ca-bundle" not found Oct 30 11:21:51.897847 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:21:51.897665 4289 projected.go:266] Couldn't get secret redhat-ods-monitoring/data-science-lokistack-gateway-http: secret "data-science-lokistack-gateway-http" not found Oct 30 11:21:51.897847 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:21:51.897689 4289 projected.go:266] Couldn't get secret redhat-ods-monitoring/data-science-lokistack-gateway-http: secret "data-science-lokistack-gateway-http" not found Oct 30 11:21:51.897847 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:21:51.897702 4289 projected.go:196] Error preparing data for projected volume tls-secret for pod redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj: secret "data-science-lokistack-gateway-http" not found Oct 30 11:21:51.897847 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:21:51.897751 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-tls-secret podName:551b0f59-4902-49a5-a83f-85bad32dee9e nodeName:}" failed. No retries permitted until 2026-10-30 11:21:52.39773469 +0000 UTC m=+2202.899114156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-secret" (UniqueName: "kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-tls-secret") pod "data-science-lokistack-gateway-698f8cbf65-btsfj" (UID: "551b0f59-4902-49a5-a83f-85bad32dee9e") : secret "data-science-lokistack-gateway-http" not found Oct 30 11:21:51.897847 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.897787 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-lokistack-gateway\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:51.898215 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.898119 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:51.898215 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.898187 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-rbac\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:51.898479 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.898461 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-lokistack-gateway\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:51.899822 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.899802 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/551b0f59-4902-49a5-a83f-85bad32dee9e-tenants\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:51.899953 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.899936 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/551b0f59-4902-49a5-a83f-85bad32dee9e-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:51.924067 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.924030 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss76v\" (UniqueName: \"kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-kube-api-access-ss76v\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:51.998944 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:51.998915 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g"] Oct 30 11:21:51.999531 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:21:51.999508 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2875bb2d_9cc3_4467_874a_bbacc4c94177.slice/crio-804099510b974ddada213f74278a5f70e77e9b1268901a94f99cccb69d2c7bec WatchSource:0}: Error finding container 804099510b974ddada213f74278a5f70e77e9b1268901a94f99cccb69d2c7bec: Status 404 returned error can't find the container with id 804099510b974ddada213f74278a5f70e77e9b1268901a94f99cccb69d2c7bec Oct 30 11:21:52.037008 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:52.036967 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:52.226996 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:52.226957 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:52.386454 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:52.386336 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" event={"ID":"2875bb2d-9cc3-4467-874a-bbacc4c94177","Type":"ContainerStarted","Data":"804099510b974ddada213f74278a5f70e77e9b1268901a94f99cccb69d2c7bec"} Oct 30 11:21:52.387523 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:52.387495 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" event={"ID":"a517cbd2-329c-4c01-8f6f-955220592158","Type":"ContainerStarted","Data":"ce2b3dbc00322ae4e9ab5d16435dffeb13f020e2c417a8234b69250b07ac75c5"} Oct 30 11:21:52.402996 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:52.402972 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-tls-secret\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:52.403085 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:52.403037 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:52.405389 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:52.405365 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-tls-secret\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:52.405505 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:52.405385 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-698f8cbf65-btsfj\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:52.669498 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:52.669361 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:53.017382 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:21:53.017339 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551b0f59_4902_49a5_a83f_85bad32dee9e.slice/crio-7ca97f1eb43afa54a74e8896793dce135cd0f6f783282cf3900221b0716dba8e WatchSource:0}: Error finding container 7ca97f1eb43afa54a74e8896793dce135cd0f6f783282cf3900221b0716dba8e: Status 404 returned error can't find the container with id 7ca97f1eb43afa54a74e8896793dce135cd0f6f783282cf3900221b0716dba8e Oct 30 11:21:53.028283 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:53.028215 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj"] Oct 30 11:21:53.401138 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:53.401006 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" event={"ID":"551b0f59-4902-49a5-a83f-85bad32dee9e","Type":"ContainerStarted","Data":"7ca97f1eb43afa54a74e8896793dce135cd0f6f783282cf3900221b0716dba8e"} Oct 30 11:21:53.751971 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:53.751878 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:53.928383 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:53.928309 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:54.521678 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:54.521569 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:54.691969 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:54.691926 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:54.955200 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:54.955083 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:55.121486 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.121412 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:55.626063 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.625989 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 11:21:55.629973 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.629947 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:21:55.645933 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.645903 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 11:21:55.655873 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.655847 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-http\"" Oct 30 11:21:55.656038 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.655984 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-grpc\"" Oct 30 11:21:55.839979 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.839935 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11866315-a378-46ca-a34a-da39ef3a8e51-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:21:55.840157 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.839983 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/11866315-a378-46ca-a34a-da39ef3a8e51-test-loki-shared-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:21:55.840157 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.840015 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/11866315-a378-46ca-a34a-da39ef3a8e51-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:21:55.840157 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.840132 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbg8f\" (UniqueName: \"kubernetes.io/projected/11866315-a378-46ca-a34a-da39ef3a8e51-kube-api-access-tbg8f\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:21:55.840311 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.840176 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11866315-a378-46ca-a34a-da39ef3a8e51-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:21:55.840311 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.840222 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/11866315-a378-46ca-a34a-da39ef3a8e51-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:21:55.842013 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.841965 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:21:55.941063 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.941037 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11866315-a378-46ca-a34a-da39ef3a8e51-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:21:55.941168 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.941075 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/11866315-a378-46ca-a34a-da39ef3a8e51-test-loki-shared-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:21:55.941168 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.941103 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/11866315-a378-46ca-a34a-da39ef3a8e51-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:21:55.941168 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.941132 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbg8f\" (UniqueName: \"kubernetes.io/projected/11866315-a378-46ca-a34a-da39ef3a8e51-kube-api-access-tbg8f\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:21:55.941168 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.941162 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11866315-a378-46ca-a34a-da39ef3a8e51-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:21:55.941377 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.941203 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/11866315-a378-46ca-a34a-da39ef3a8e51-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:21:55.941858 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.941836 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11866315-a378-46ca-a34a-da39ef3a8e51-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:21:55.941989 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.941929 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11866315-a378-46ca-a34a-da39ef3a8e51-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:21:55.943905 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.943872 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/11866315-a378-46ca-a34a-da39ef3a8e51-test-loki-shared-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:21:55.943992 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.943958 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/11866315-a378-46ca-a34a-da39ef3a8e51-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:21:55.944113 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.944090 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/11866315-a378-46ca-a34a-da39ef3a8e51-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:21:55.963847 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:55.963808 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbg8f\" (UniqueName: \"kubernetes.io/projected/11866315-a378-46ca-a34a-da39ef3a8e51-kube-api-access-tbg8f\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:21:56.260918 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:21:56.260883 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:21:56.420112 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:56.420079 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" event={"ID":"a517cbd2-329c-4c01-8f6f-955220592158","Type":"ContainerStarted","Data":"1fb3677247ef00a9b02ecebc668da95486722d4219c5592fe4180c0da05d8bcf"} Oct 30 11:21:56.420281 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:56.420238 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" Oct 30 11:21:56.421776 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:56.421753 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" event={"ID":"551b0f59-4902-49a5-a83f-85bad32dee9e","Type":"ContainerStarted","Data":"c64a94339ad91ba61a9f913009603b94e1b43e788c3e533297445587da582c6c"} Oct 30 11:21:56.423221 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:56.423200 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" event={"ID":"2875bb2d-9cc3-4467-874a-bbacc4c94177","Type":"ContainerStarted","Data":"55498903d79bcfcd9b086eebd52b586ebf11c6a5b3109008afa5a2f7beff9dfe"} Oct 30 11:21:56.423295 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:56.423249 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" Oct 30 11:21:56.461895 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:56.461836 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" podStartSLOduration=1.3317428709999999 podStartE2EDuration="5.461818721s" totalImagesPullingTime="4.13007585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:21:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:21:56.46123002 +0000 UTC m=+2206.962609505" watchObservedRunningTime="2026-10-30 11:21:56.461818721 +0000 UTC m=+2206.963198208" Oct 30 11:21:57.430017 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:57.429980 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" event={"ID":"551b0f59-4902-49a5-a83f-85bad32dee9e","Type":"ContainerStarted","Data":"308a55367339f15b085527624420d6c6479a31b22e52a52de4d0274c0bc51c91"} Oct 30 11:21:57.430539 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:57.430226 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:57.430539 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:57.430281 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:57.436569 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:57.436549 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:57.439066 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:57.439045 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:21:57.499255 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:57.499203 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" podStartSLOduration=2.523528703 podStartE2EDuration="6.499190018s" totalImagesPullingTime="3.975661315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:21:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:21:56.505499614 +0000 UTC m=+2207.006879100" watchObservedRunningTime="2026-10-30 11:21:57.499190018 +0000 UTC m=+2208.000569496" Oct 30 11:21:57.499608 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:57.499585 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" podStartSLOduration=2.856378366 podStartE2EDuration="6.499576201s" totalImagesPullingTime="3.643197835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:21:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:21:57.498105856 +0000 UTC m=+2207.999485373" watchObservedRunningTime="2026-10-30 11:21:57.499576201 +0000 UTC m=+2208.000955694" Oct 30 11:21:57.660144 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:21:57.660062 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:22:00.078915 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:22:00.078879 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:22:00.106939 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:22:00.106911 4289 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:22:00.179263 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:22:00.179233 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:22:00.179473 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:22:00.179374 4289 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:22:00.193425 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:22:00.193398 4289 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") DevicePath \"csi-2e669b6d2bbdfa3a6b46036bf22d4a7a6aba8888c2c130325f5aeb55a396d7c5\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:22:01.362683 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:22:01.362652 4289 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b48470aea0a0a5b9ac05ab05099d84e77041b5209a87fa2cabb2ba39e3c57c0c/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:22:01.388779 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:22:01.388749 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:22:01.646755 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:22:01.646679 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:22:01.942213 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:22:01.942183 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 11:22:01.942456 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:22:01.942409 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11866315_a378_46ca_a34a_da39ef3a8e51.slice/crio-6d9fc55074054900b0238143edcb8e920b9586af5b80e00ae27520073b84140b WatchSource:0}: Error finding container 6d9fc55074054900b0238143edcb8e920b9586af5b80e00ae27520073b84140b: Status 404 returned error can't find the container with id 6d9fc55074054900b0238143edcb8e920b9586af5b80e00ae27520073b84140b Oct 30 11:22:02.455621 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:22:02.455584 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"11866315-a378-46ca-a34a-da39ef3a8e51","Type":"ContainerStarted","Data":"f8f8cccc51cbe4e873ebe8d9718db06b9d7ed73357bacb2af466031d12dd95ee"} Oct 30 11:22:02.456024 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:22:02.455632 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"11866315-a378-46ca-a34a-da39ef3a8e51","Type":"ContainerStarted","Data":"6d9fc55074054900b0238143edcb8e920b9586af5b80e00ae27520073b84140b"} Oct 30 11:22:02.456024 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:22:02.455713 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:22:02.499101 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:22:02.499055 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" podStartSLOduration=11.499042204 podStartE2EDuration="11.499042204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:22:02.498259498 +0000 UTC m=+2212.999638984" watchObservedRunningTime="2026-10-30 11:22:02.499042204 +0000 UTC m=+2213.000421688" Oct 30 11:22:17.436518 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:22:17.436488 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" Oct 30 11:22:17.437009 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:22:17.436565 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" Oct 30 11:22:23.466458 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:22:23.466417 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:22:23.644875 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:22:23.644844 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:22:23.805237 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:22:23.805193 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:22:25.913020 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:22:25.912979 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:22:26.094699 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:22:26.094666 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:22:26.258257 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:22:26.258220 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:22:26.789579 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:22:26.789543 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:22:56.257767 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:22:56.257730 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:22:57.800138 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:22:57.800104 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:22:58.629749 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:22:58.629719 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:22:58.794147 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:22:58.794089 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:23:00.910002 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:00.909960 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:23:01.080729 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.080696 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:23:01.582677 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.582644 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"] Oct 30 11:23:01.585297 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.585281 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:01.613686 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.613662 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-http\"" Oct 30 11:23:01.613817 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.613712 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-grpc\"" Oct 30 11:23:01.625206 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.625184 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"] Oct 30 11:23:01.791636 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.791584 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-config\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:01.791814 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.791691 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:01.791814 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.791712 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:01.791814 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.791748 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:01.791814 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.791808 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-test-loki-shared-secret\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:01.791966 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.791858 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzjwk\" (UniqueName: \"kubernetes.io/projected/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-kube-api-access-mzjwk\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:01.815557 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.815516 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:23:01.893335 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.893259 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzjwk\" (UniqueName: \"kubernetes.io/projected/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-kube-api-access-mzjwk\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:01.893335 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.893314 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-config\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:01.893533 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.893340 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:01.893533 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.893356 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:01.893533 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.893376 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:01.893533 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.893403 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-test-loki-shared-secret\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:01.894111 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.894081 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:01.894244 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.894197 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-config\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:01.895972 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.895953 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:01.896082 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.896016 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:01.896278 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.896258 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-test-loki-shared-secret\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:01.916901 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:01.916879 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzjwk\" (UniqueName: \"kubernetes.io/projected/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-kube-api-access-mzjwk\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:11.576372 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:11.576330 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-5016d491-1af7-4119-a668-95a1697a696b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:11.576826 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:11.576393 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6dcdb7ad-d1ff-4871-aad9-1c2bebcc5bb8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:11.605008 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:11.604976 4289 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-5016d491-1af7-4119-a668-95a1697a696b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:11.605673 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:11.605651 4289 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6dcdb7ad-d1ff-4871-aad9-1c2bebcc5bb8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:11.677858 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:11.677827 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-5016d491-1af7-4119-a668-95a1697a696b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:11.678037 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:11.677876 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6dcdb7ad-d1ff-4871-aad9-1c2bebcc5bb8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:11.678037 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:11.677975 4289 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6dcdb7ad-d1ff-4871-aad9-1c2bebcc5bb8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:11.678037 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:11.677997 4289 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-5016d491-1af7-4119-a668-95a1697a696b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:11.692810 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:11.692786 4289 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-5016d491-1af7-4119-a668-95a1697a696b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") DevicePath \"csi-24162340a6be733c56063f2faeaf5eac3653e9c6e6136a764bd2aa7236d709ba\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:11.692895 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:11.692785 4289 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6dcdb7ad-d1ff-4871-aad9-1c2bebcc5bb8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") DevicePath \"csi-fed6c8c359d982cd671ddd7b8904c0992b96b06853cad388882421bf5eda486c\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:11.925131 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:11.925092 4289 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6dcdb7ad-d1ff-4871-aad9-1c2bebcc5bb8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/186c84c83760be5837750810e291c0bfee785cbd5ee4b0582f60daafa9018035/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:11.951663 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:11.951631 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6dcdb7ad-d1ff-4871-aad9-1c2bebcc5bb8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:19.015984 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:19.015949 4289 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-5016d491-1af7-4119-a668-95a1697a696b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/0d90e1e43de51125033a7f82cd0645765bf678ea2057dc8845fad0d0b4c35b0f/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:19.041950 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:19.041909 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-5016d491-1af7-4119-a668-95a1697a696b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3\") pod \"data-science-lokistack-ingester-1\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:19.296040 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:19.295958 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:19.632858 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:19.627338 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"] Oct 30 11:23:19.780577 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:19.780539 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" event={"ID":"5976f8fd-b82f-492c-93e2-f0b01f0d9e60","Type":"ContainerStarted","Data":"5648fa4c64daf07346c3856878e2da64cc2a2712e42dfb882d5dbcea9df63cba"} Oct 30 11:23:19.780750 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:19.780596 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" event={"ID":"5976f8fd-b82f-492c-93e2-f0b01f0d9e60","Type":"ContainerStarted","Data":"d69921d0c5ae80c18c8db6a5ec15ab4a318db17c32c1db10ff0e29be6b819764"} Oct 30 11:23:19.780750 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:19.780688 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:23:19.834312 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:19.834261 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podStartSLOduration=21.834246654 podStartE2EDuration="21.834246654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:23:19.832664603 +0000 UTC m=+2290.334044089" watchObservedRunningTime="2026-10-30 11:23:19.834246654 +0000 UTC m=+2290.335626139" Oct 30 11:23:26.257194 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:23:26.257107 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:23:40.791885 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:40.791847 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podUID="5976f8fd-b82f-492c-93e2-f0b01f0d9e60" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 11:23:50.790090 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:50.790054 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podUID="5976f8fd-b82f-492c-93e2-f0b01f0d9e60" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 11:23:56.257455 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:23:56.257402 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:23:57.932035 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:23:57.932003 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:24:00.790544 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:24:00.790502 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podUID="5976f8fd-b82f-492c-93e2-f0b01f0d9e60" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 11:24:10.790373 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:24:10.790340 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:24:10.975237 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:24:10.975204 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:24:11.152806 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:24:11.152705 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:24:13.286872 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:24:13.286835 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:24:13.454346 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:24:13.454306 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:24:14.111856 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:24:14.111806 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:24:14.276741 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:24:14.276708 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:24:26.257584 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:24:26.257546 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:24:56.257380 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:24:56.257336 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:24:58.027084 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:24:58.027043 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:24:58.975564 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:24:58.975527 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:25:26.256884 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:25:26.256844 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:25:30.911017 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:30.910982 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-697dbdbd58-qpwvl"] Oct 30 11:25:30.916369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:30.916345 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-697dbdbd58-qpwvl" Oct 30 11:25:30.936762 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:30.936731 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-697dbdbd58-qpwvl"] Oct 30 11:25:30.946775 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:30.946752 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-dockercfg-tk7xf\"" Oct 30 11:25:30.962830 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:30.962801 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:25:30.975645 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:30.975605 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4e11ba87-4e5a-407d-8f1d-28d8ec59ab25-otc-internal\") pod \"data-science-usage-logs-collector-697dbdbd58-qpwvl\" (UID: \"4e11ba87-4e5a-407d-8f1d-28d8ec59ab25\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-697dbdbd58-qpwvl" Oct 30 11:25:30.975779 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:30.975759 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5vtc\" (UniqueName: \"kubernetes.io/projected/4e11ba87-4e5a-407d-8f1d-28d8ec59ab25-kube-api-access-l5vtc\") pod \"data-science-usage-logs-collector-697dbdbd58-qpwvl\" (UID: \"4e11ba87-4e5a-407d-8f1d-28d8ec59ab25\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-697dbdbd58-qpwvl" Oct 30 11:25:31.006307 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:31.006278 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-6a6878d3\"" Oct 30 11:25:31.077063 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:31.077029 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5vtc\" (UniqueName: \"kubernetes.io/projected/4e11ba87-4e5a-407d-8f1d-28d8ec59ab25-kube-api-access-l5vtc\") pod \"data-science-usage-logs-collector-697dbdbd58-qpwvl\" (UID: \"4e11ba87-4e5a-407d-8f1d-28d8ec59ab25\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-697dbdbd58-qpwvl" Oct 30 11:25:31.077258 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:31.077102 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4e11ba87-4e5a-407d-8f1d-28d8ec59ab25-otc-internal\") pod \"data-science-usage-logs-collector-697dbdbd58-qpwvl\" (UID: \"4e11ba87-4e5a-407d-8f1d-28d8ec59ab25\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-697dbdbd58-qpwvl" Oct 30 11:25:31.077707 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:31.077686 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4e11ba87-4e5a-407d-8f1d-28d8ec59ab25-otc-internal\") pod \"data-science-usage-logs-collector-697dbdbd58-qpwvl\" (UID: \"4e11ba87-4e5a-407d-8f1d-28d8ec59ab25\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-697dbdbd58-qpwvl" Oct 30 11:25:31.100650 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:31.100627 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5vtc\" (UniqueName: \"kubernetes.io/projected/4e11ba87-4e5a-407d-8f1d-28d8ec59ab25-kube-api-access-l5vtc\") pod \"data-science-usage-logs-collector-697dbdbd58-qpwvl\" (UID: \"4e11ba87-4e5a-407d-8f1d-28d8ec59ab25\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-697dbdbd58-qpwvl" Oct 30 11:25:31.139751 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:31.139726 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:25:31.231771 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:31.231746 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-697dbdbd58-qpwvl" Oct 30 11:25:31.524087 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:31.524058 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-697dbdbd58-qpwvl"] Oct 30 11:25:31.524517 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:25:31.524495 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e11ba87_4e5a_407d_8f1d_28d8ec59ab25.slice/crio-bfc24feeec5f0a762c0de852e112768ea31ec9cd1dca22c2a52c3db01de8d95e WatchSource:0}: Error finding container bfc24feeec5f0a762c0de852e112768ea31ec9cd1dca22c2a52c3db01de8d95e: Status 404 returned error can't find the container with id bfc24feeec5f0a762c0de852e112768ea31ec9cd1dca22c2a52c3db01de8d95e Oct 30 11:25:31.526599 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:31.526566 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:25:31.526714 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:31.526618 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:25:32.351337 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:32.351301 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-697dbdbd58-qpwvl" event={"ID":"4e11ba87-4e5a-407d-8f1d-28d8ec59ab25","Type":"ContainerStarted","Data":"a52ab3074f9e94d2c8dbbbd46ea7591bac873a004ebe87989fe5171251d6927f"} Oct 30 11:25:32.351841 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:32.351359 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-697dbdbd58-qpwvl" event={"ID":"4e11ba87-4e5a-407d-8f1d-28d8ec59ab25","Type":"ContainerStarted","Data":"bfc24feeec5f0a762c0de852e112768ea31ec9cd1dca22c2a52c3db01de8d95e"} Oct 30 11:25:32.392315 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:32.392256 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-697dbdbd58-qpwvl" podStartSLOduration=2.392242632 podStartE2EDuration="2.392242632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:25:32.390675158 +0000 UTC m=+2422.892054668" watchObservedRunningTime="2026-10-30 11:25:32.392242632 +0000 UTC m=+2422.893622117" Oct 30 11:25:34.668660 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:34.668628 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5896c48b7-qnsnw"] Oct 30 11:25:34.671876 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:34.671861 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5896c48b7-qnsnw" Oct 30 11:25:34.691524 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:34.691498 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5896c48b7-qnsnw"] Oct 30 11:25:34.812092 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:34.812055 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk5d6\" (UniqueName: \"kubernetes.io/projected/98f32b11-4f0d-4afb-b45a-e481dc3453e5-kube-api-access-sk5d6\") pod \"data-science-usage-logs-collector-5896c48b7-qnsnw\" (UID: \"98f32b11-4f0d-4afb-b45a-e481dc3453e5\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5896c48b7-qnsnw" Oct 30 11:25:34.812255 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:34.812193 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/98f32b11-4f0d-4afb-b45a-e481dc3453e5-otc-internal\") pod \"data-science-usage-logs-collector-5896c48b7-qnsnw\" (UID: \"98f32b11-4f0d-4afb-b45a-e481dc3453e5\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5896c48b7-qnsnw" Oct 30 11:25:34.829469 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:34.829413 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:25:34.913523 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:34.913494 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk5d6\" (UniqueName: \"kubernetes.io/projected/98f32b11-4f0d-4afb-b45a-e481dc3453e5-kube-api-access-sk5d6\") pod \"data-science-usage-logs-collector-5896c48b7-qnsnw\" (UID: \"98f32b11-4f0d-4afb-b45a-e481dc3453e5\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5896c48b7-qnsnw" Oct 30 11:25:34.913675 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:34.913565 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/98f32b11-4f0d-4afb-b45a-e481dc3453e5-otc-internal\") pod \"data-science-usage-logs-collector-5896c48b7-qnsnw\" (UID: \"98f32b11-4f0d-4afb-b45a-e481dc3453e5\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5896c48b7-qnsnw" Oct 30 11:25:34.914172 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:34.914148 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/98f32b11-4f0d-4afb-b45a-e481dc3453e5-otc-internal\") pod \"data-science-usage-logs-collector-5896c48b7-qnsnw\" (UID: \"98f32b11-4f0d-4afb-b45a-e481dc3453e5\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5896c48b7-qnsnw" Oct 30 11:25:34.936638 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:34.936577 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk5d6\" (UniqueName: \"kubernetes.io/projected/98f32b11-4f0d-4afb-b45a-e481dc3453e5-kube-api-access-sk5d6\") pod \"data-science-usage-logs-collector-5896c48b7-qnsnw\" (UID: \"98f32b11-4f0d-4afb-b45a-e481dc3453e5\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5896c48b7-qnsnw" Oct 30 11:25:34.984178 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:34.984150 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5896c48b7-qnsnw" Oct 30 11:25:35.009347 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:35.009314 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:25:35.280758 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:25:35.280714 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f32b11_4f0d_4afb_b45a_e481dc3453e5.slice/crio-975c7e648cbb0531920d82aab705c95205ca4c74141f7df36c0357b2955283d1 WatchSource:0}: Error finding container 975c7e648cbb0531920d82aab705c95205ca4c74141f7df36c0357b2955283d1: Status 404 returned error can't find the container with id 975c7e648cbb0531920d82aab705c95205ca4c74141f7df36c0357b2955283d1 Oct 30 11:25:35.280893 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:35.280761 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5896c48b7-qnsnw"] Oct 30 11:25:35.283005 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:35.282973 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:25:35.283113 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:35.283036 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:25:35.372177 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:35.372148 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5896c48b7-qnsnw" event={"ID":"98f32b11-4f0d-4afb-b45a-e481dc3453e5","Type":"ContainerStarted","Data":"975c7e648cbb0531920d82aab705c95205ca4c74141f7df36c0357b2955283d1"} Oct 30 11:25:36.378017 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:36.377980 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5896c48b7-qnsnw" event={"ID":"98f32b11-4f0d-4afb-b45a-e481dc3453e5","Type":"ContainerStarted","Data":"64554cfb379e88d6a0d66ddd793986a4375af2bb8d3e69f45df99a8a66e8700a"} Oct 30 11:25:36.394419 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:36.394390 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:25:36.418394 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:36.418347 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5896c48b7-qnsnw" podStartSLOduration=2.418335475 podStartE2EDuration="2.418335475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:25:36.417594597 +0000 UTC m=+2426.918974082" watchObservedRunningTime="2026-10-30 11:25:36.418335475 +0000 UTC m=+2426.919714962" Oct 30 11:25:36.449585 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:36.449554 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-697dbdbd58-qpwvl"] Oct 30 11:25:36.449854 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:36.449829 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-697dbdbd58-qpwvl" podUID="4e11ba87-4e5a-407d-8f1d-28d8ec59ab25" containerName="otc-container" containerID="cri-o://a52ab3074f9e94d2c8dbbbd46ea7591bac873a004ebe87989fe5171251d6927f" gracePeriod=30 Oct 30 11:25:36.556842 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:36.556804 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:25:36.719447 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:36.719414 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-697dbdbd58-qpwvl" Oct 30 11:25:36.839326 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:36.839297 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e11ba87-4e5a-407d-8f1d-28d8ec59ab25-kube-api-access-l5vtc\" (UniqueName: \"kubernetes.io/projected/4e11ba87-4e5a-407d-8f1d-28d8ec59ab25-kube-api-access-l5vtc\") pod \"4e11ba87-4e5a-407d-8f1d-28d8ec59ab25\" (UID: \"4e11ba87-4e5a-407d-8f1d-28d8ec59ab25\") " Oct 30 11:25:36.839540 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:36.839412 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e11ba87-4e5a-407d-8f1d-28d8ec59ab25-otc-internal\" (UniqueName: \"kubernetes.io/configmap/4e11ba87-4e5a-407d-8f1d-28d8ec59ab25-otc-internal\") pod \"4e11ba87-4e5a-407d-8f1d-28d8ec59ab25\" (UID: \"4e11ba87-4e5a-407d-8f1d-28d8ec59ab25\") " Oct 30 11:25:36.839752 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:36.839731 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e11ba87-4e5a-407d-8f1d-28d8ec59ab25-otc-internal" pod "4e11ba87-4e5a-407d-8f1d-28d8ec59ab25" (UID: "4e11ba87-4e5a-407d-8f1d-28d8ec59ab25"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:25:36.841486 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:36.841462 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e11ba87-4e5a-407d-8f1d-28d8ec59ab25-kube-api-access-l5vtc" pod "4e11ba87-4e5a-407d-8f1d-28d8ec59ab25" (UID: "4e11ba87-4e5a-407d-8f1d-28d8ec59ab25"). InnerVolumeSpecName "kube-api-access-l5vtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:25:36.940980 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:36.940948 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5vtc\" (UniqueName: \"kubernetes.io/projected/4e11ba87-4e5a-407d-8f1d-28d8ec59ab25-kube-api-access-l5vtc\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:25:36.940980 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:36.940974 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4e11ba87-4e5a-407d-8f1d-28d8ec59ab25-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:25:37.339356 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:37.339324 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5896c48b7-qnsnw"] Oct 30 11:25:37.383879 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:37.383844 4289 generic.go:356] "Generic (PLEG): container finished" podID="4e11ba87-4e5a-407d-8f1d-28d8ec59ab25" containerID="a52ab3074f9e94d2c8dbbbd46ea7591bac873a004ebe87989fe5171251d6927f" exitCode=0 Oct 30 11:25:37.384309 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:37.383892 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-697dbdbd58-qpwvl" Oct 30 11:25:37.384309 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:37.383929 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-697dbdbd58-qpwvl" event={"ID":"4e11ba87-4e5a-407d-8f1d-28d8ec59ab25","Type":"ContainerDied","Data":"a52ab3074f9e94d2c8dbbbd46ea7591bac873a004ebe87989fe5171251d6927f"} Oct 30 11:25:37.384309 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:37.383987 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-697dbdbd58-qpwvl" event={"ID":"4e11ba87-4e5a-407d-8f1d-28d8ec59ab25","Type":"ContainerDied","Data":"bfc24feeec5f0a762c0de852e112768ea31ec9cd1dca22c2a52c3db01de8d95e"} Oct 30 11:25:37.384309 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:37.384014 4289 scope.go:122] "RemoveContainer" containerID="a52ab3074f9e94d2c8dbbbd46ea7591bac873a004ebe87989fe5171251d6927f" Oct 30 11:25:37.384522 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:37.384359 4289 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/data-science-usage-logs-collector-5896c48b7-qnsnw" secret="" err="secret \"data-science-usage-logs-collector-dockercfg-tk7xf\" not found" Oct 30 11:25:37.393689 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:37.393664 4289 scope.go:122] "RemoveContainer" containerID="a52ab3074f9e94d2c8dbbbd46ea7591bac873a004ebe87989fe5171251d6927f" Oct 30 11:25:37.393993 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:25:37.393975 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a52ab3074f9e94d2c8dbbbd46ea7591bac873a004ebe87989fe5171251d6927f\": container with ID starting with a52ab3074f9e94d2c8dbbbd46ea7591bac873a004ebe87989fe5171251d6927f not found: ID does not exist" containerID="a52ab3074f9e94d2c8dbbbd46ea7591bac873a004ebe87989fe5171251d6927f" Oct 30 11:25:37.394080 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:37.394002 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a52ab3074f9e94d2c8dbbbd46ea7591bac873a004ebe87989fe5171251d6927f"} err="failed to get container status \"a52ab3074f9e94d2c8dbbbd46ea7591bac873a004ebe87989fe5171251d6927f\": rpc error: code = NotFound desc = could not find container \"a52ab3074f9e94d2c8dbbbd46ea7591bac873a004ebe87989fe5171251d6927f\": container with ID starting with a52ab3074f9e94d2c8dbbbd46ea7591bac873a004ebe87989fe5171251d6927f not found: ID does not exist" Oct 30 11:25:37.422504 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:37.422475 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:25:37.434655 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:37.434626 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-697dbdbd58-qpwvl"] Oct 30 11:25:37.438137 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:37.438112 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-697dbdbd58-qpwvl"] Oct 30 11:25:37.546290 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:25:37.546258 4289 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-usage-logs-collector-6a6878d3: configmap "data-science-usage-logs-collector-6a6878d3" not found Oct 30 11:25:37.546468 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:25:37.546344 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98f32b11-4f0d-4afb-b45a-e481dc3453e5-otc-internal podName:98f32b11-4f0d-4afb-b45a-e481dc3453e5 nodeName:}" failed. No retries permitted until 2026-10-30 11:25:38.046323949 +0000 UTC m=+2428.547703428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/98f32b11-4f0d-4afb-b45a-e481dc3453e5-otc-internal") pod "data-science-usage-logs-collector-5896c48b7-qnsnw" (UID: "98f32b11-4f0d-4afb-b45a-e481dc3453e5") : configmap "data-science-usage-logs-collector-6a6878d3" not found Oct 30 11:25:37.597467 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:37.597351 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:25:37.625607 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:37.625576 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e11ba87-4e5a-407d-8f1d-28d8ec59ab25" path="/var/lib/kubelet/pods/4e11ba87-4e5a-407d-8f1d-28d8ec59ab25/volumes" Oct 30 11:25:38.050917 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:25:38.050886 4289 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-usage-logs-collector-6a6878d3: configmap "data-science-usage-logs-collector-6a6878d3" not found Oct 30 11:25:38.051099 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:25:38.050968 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98f32b11-4f0d-4afb-b45a-e481dc3453e5-otc-internal podName:98f32b11-4f0d-4afb-b45a-e481dc3453e5 nodeName:}" failed. No retries permitted until 2026-10-30 11:25:39.050949047 +0000 UTC m=+2429.552328515 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/98f32b11-4f0d-4afb-b45a-e481dc3453e5-otc-internal") pod "data-science-usage-logs-collector-5896c48b7-qnsnw" (UID: "98f32b11-4f0d-4afb-b45a-e481dc3453e5") : configmap "data-science-usage-logs-collector-6a6878d3" not found Oct 30 11:25:38.389166 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:38.389046 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5896c48b7-qnsnw" podUID="98f32b11-4f0d-4afb-b45a-e481dc3453e5" containerName="otc-container" containerID="cri-o://64554cfb379e88d6a0d66ddd793986a4375af2bb8d3e69f45df99a8a66e8700a" gracePeriod=30 Oct 30 11:25:38.665045 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:38.665021 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5896c48b7-qnsnw" Oct 30 11:25:38.758010 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:38.757976 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98f32b11-4f0d-4afb-b45a-e481dc3453e5-kube-api-access-sk5d6\" (UniqueName: \"kubernetes.io/projected/98f32b11-4f0d-4afb-b45a-e481dc3453e5-kube-api-access-sk5d6\") pod \"98f32b11-4f0d-4afb-b45a-e481dc3453e5\" (UID: \"98f32b11-4f0d-4afb-b45a-e481dc3453e5\") " Oct 30 11:25:38.758202 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:38.758043 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98f32b11-4f0d-4afb-b45a-e481dc3453e5-otc-internal\" (UniqueName: \"kubernetes.io/configmap/98f32b11-4f0d-4afb-b45a-e481dc3453e5-otc-internal\") pod \"98f32b11-4f0d-4afb-b45a-e481dc3453e5\" (UID: \"98f32b11-4f0d-4afb-b45a-e481dc3453e5\") " Oct 30 11:25:38.758376 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:38.758354 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f32b11-4f0d-4afb-b45a-e481dc3453e5-otc-internal" pod "98f32b11-4f0d-4afb-b45a-e481dc3453e5" (UID: "98f32b11-4f0d-4afb-b45a-e481dc3453e5"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:25:38.760051 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:38.760022 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f32b11-4f0d-4afb-b45a-e481dc3453e5-kube-api-access-sk5d6" pod "98f32b11-4f0d-4afb-b45a-e481dc3453e5" (UID: "98f32b11-4f0d-4afb-b45a-e481dc3453e5"). InnerVolumeSpecName "kube-api-access-sk5d6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:25:38.858666 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:38.858637 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk5d6\" (UniqueName: \"kubernetes.io/projected/98f32b11-4f0d-4afb-b45a-e481dc3453e5-kube-api-access-sk5d6\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:25:38.858666 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:38.858664 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/98f32b11-4f0d-4afb-b45a-e481dc3453e5-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:25:39.395150 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:39.395113 4289 generic.go:356] "Generic (PLEG): container finished" podID="98f32b11-4f0d-4afb-b45a-e481dc3453e5" containerID="64554cfb379e88d6a0d66ddd793986a4375af2bb8d3e69f45df99a8a66e8700a" exitCode=0 Oct 30 11:25:39.395665 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:39.395164 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5896c48b7-qnsnw" Oct 30 11:25:39.395665 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:39.395197 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5896c48b7-qnsnw" event={"ID":"98f32b11-4f0d-4afb-b45a-e481dc3453e5","Type":"ContainerDied","Data":"64554cfb379e88d6a0d66ddd793986a4375af2bb8d3e69f45df99a8a66e8700a"} Oct 30 11:25:39.395665 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:39.395265 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5896c48b7-qnsnw" event={"ID":"98f32b11-4f0d-4afb-b45a-e481dc3453e5","Type":"ContainerDied","Data":"975c7e648cbb0531920d82aab705c95205ca4c74141f7df36c0357b2955283d1"} Oct 30 11:25:39.395665 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:39.395297 4289 scope.go:122] "RemoveContainer" containerID="64554cfb379e88d6a0d66ddd793986a4375af2bb8d3e69f45df99a8a66e8700a" Oct 30 11:25:39.404075 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:39.404056 4289 scope.go:122] "RemoveContainer" containerID="64554cfb379e88d6a0d66ddd793986a4375af2bb8d3e69f45df99a8a66e8700a" Oct 30 11:25:39.404301 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:25:39.404283 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64554cfb379e88d6a0d66ddd793986a4375af2bb8d3e69f45df99a8a66e8700a\": container with ID starting with 64554cfb379e88d6a0d66ddd793986a4375af2bb8d3e69f45df99a8a66e8700a not found: ID does not exist" containerID="64554cfb379e88d6a0d66ddd793986a4375af2bb8d3e69f45df99a8a66e8700a" Oct 30 11:25:39.404368 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:39.404324 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64554cfb379e88d6a0d66ddd793986a4375af2bb8d3e69f45df99a8a66e8700a"} err="failed to get container status \"64554cfb379e88d6a0d66ddd793986a4375af2bb8d3e69f45df99a8a66e8700a\": rpc error: code = NotFound desc = could not find container \"64554cfb379e88d6a0d66ddd793986a4375af2bb8d3e69f45df99a8a66e8700a\": container with ID starting with 64554cfb379e88d6a0d66ddd793986a4375af2bb8d3e69f45df99a8a66e8700a not found: ID does not exist" Oct 30 11:25:39.444659 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:39.444635 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5896c48b7-qnsnw"] Oct 30 11:25:39.456623 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:39.456601 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5896c48b7-qnsnw"] Oct 30 11:25:39.624645 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:39.624616 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98f32b11-4f0d-4afb-b45a-e481dc3453e5" path="/var/lib/kubelet/pods/98f32b11-4f0d-4afb-b45a-e481dc3453e5/volumes" Oct 30 11:25:41.344952 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:41.344910 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:25:46.954527 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:46.954485 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:25:56.258047 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:25:56.258010 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:25:56.608481 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.608387 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g"] Oct 30 11:25:56.608715 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.608664 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" podUID="2875bb2d-9cc3-4467-874a-bbacc4c94177" containerName="loki-querier" containerID="cri-o://55498903d79bcfcd9b086eebd52b586ebf11c6a5b3109008afa5a2f7beff9dfe" gracePeriod=30 Oct 30 11:25:56.661803 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.661770 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z"] Oct 30 11:25:56.662105 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.662094 4289 cpu_manager.go:414] "containerMap: removing container" podUID="4e11ba87-4e5a-407d-8f1d-28d8ec59ab25" containerName="otc-container" Oct 30 11:25:56.662151 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.662107 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e11ba87-4e5a-407d-8f1d-28d8ec59ab25" containerName="otc-container" Oct 30 11:25:56.662151 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.662126 4289 cpu_manager.go:414] "containerMap: removing container" podUID="98f32b11-4f0d-4afb-b45a-e481dc3453e5" containerName="otc-container" Oct 30 11:25:56.662151 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.662131 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98f32b11-4f0d-4afb-b45a-e481dc3453e5" containerName="otc-container" Oct 30 11:25:56.662239 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.662189 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="98f32b11-4f0d-4afb-b45a-e481dc3453e5" containerName="otc-container" Oct 30 11:25:56.662239 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.662201 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e11ba87-4e5a-407d-8f1d-28d8ec59ab25" containerName="otc-container" Oct 30 11:25:56.666348 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.666333 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" Oct 30 11:25:56.680785 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.680762 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-lifecycle-secret\"" Oct 30 11:25:56.686200 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.686173 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z"] Oct 30 11:25:56.705334 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.705304 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11f6dbe1-349f-4ebb-b891-82e63db6d59a-config\") pod \"data-science-lokistack-querier-56ffbd5fdc-6kc8z\" (UID: \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" Oct 30 11:25:56.705549 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.705347 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/11f6dbe1-349f-4ebb-b891-82e63db6d59a-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-56ffbd5fdc-6kc8z\" (UID: \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" Oct 30 11:25:56.705549 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.705413 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11f6dbe1-349f-4ebb-b891-82e63db6d59a-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-56ffbd5fdc-6kc8z\" (UID: \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" Oct 30 11:25:56.705688 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.705544 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/11f6dbe1-349f-4ebb-b891-82e63db6d59a-test-loki-lifecycle-secret\") pod \"data-science-lokistack-querier-56ffbd5fdc-6kc8z\" (UID: \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" Oct 30 11:25:56.705688 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.705573 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v6jk\" (UniqueName: \"kubernetes.io/projected/11f6dbe1-349f-4ebb-b891-82e63db6d59a-kube-api-access-7v6jk\") pod \"data-science-lokistack-querier-56ffbd5fdc-6kc8z\" (UID: \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" Oct 30 11:25:56.705688 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.705610 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/11f6dbe1-349f-4ebb-b891-82e63db6d59a-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-56ffbd5fdc-6kc8z\" (UID: \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" Oct 30 11:25:56.806348 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.806313 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11f6dbe1-349f-4ebb-b891-82e63db6d59a-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-56ffbd5fdc-6kc8z\" (UID: \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" Oct 30 11:25:56.806532 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.806373 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/11f6dbe1-349f-4ebb-b891-82e63db6d59a-test-loki-lifecycle-secret\") pod \"data-science-lokistack-querier-56ffbd5fdc-6kc8z\" (UID: \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" Oct 30 11:25:56.806532 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.806392 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v6jk\" (UniqueName: \"kubernetes.io/projected/11f6dbe1-349f-4ebb-b891-82e63db6d59a-kube-api-access-7v6jk\") pod \"data-science-lokistack-querier-56ffbd5fdc-6kc8z\" (UID: \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" Oct 30 11:25:56.806532 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.806412 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/11f6dbe1-349f-4ebb-b891-82e63db6d59a-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-56ffbd5fdc-6kc8z\" (UID: \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" Oct 30 11:25:56.806694 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.806552 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11f6dbe1-349f-4ebb-b891-82e63db6d59a-config\") pod \"data-science-lokistack-querier-56ffbd5fdc-6kc8z\" (UID: \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" Oct 30 11:25:56.806694 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.806593 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/11f6dbe1-349f-4ebb-b891-82e63db6d59a-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-56ffbd5fdc-6kc8z\" (UID: \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" Oct 30 11:25:56.807051 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.807022 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11f6dbe1-349f-4ebb-b891-82e63db6d59a-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-56ffbd5fdc-6kc8z\" (UID: \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" Oct 30 11:25:56.807402 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.807374 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11f6dbe1-349f-4ebb-b891-82e63db6d59a-config\") pod \"data-science-lokistack-querier-56ffbd5fdc-6kc8z\" (UID: \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" Oct 30 11:25:56.808987 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.808964 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/11f6dbe1-349f-4ebb-b891-82e63db6d59a-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-56ffbd5fdc-6kc8z\" (UID: \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" Oct 30 11:25:56.809076 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.809006 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/11f6dbe1-349f-4ebb-b891-82e63db6d59a-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-56ffbd5fdc-6kc8z\" (UID: \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" Oct 30 11:25:56.809468 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.809420 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/11f6dbe1-349f-4ebb-b891-82e63db6d59a-test-loki-lifecycle-secret\") pod \"data-science-lokistack-querier-56ffbd5fdc-6kc8z\" (UID: \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" Oct 30 11:25:56.829321 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.829296 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v6jk\" (UniqueName: \"kubernetes.io/projected/11f6dbe1-349f-4ebb-b891-82e63db6d59a-kube-api-access-7v6jk\") pod \"data-science-lokistack-querier-56ffbd5fdc-6kc8z\" (UID: \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" Oct 30 11:25:56.977739 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:56.977710 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" Oct 30 11:25:57.278942 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:25:57.278906 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f6dbe1_349f_4ebb_b891_82e63db6d59a.slice/crio-66ba9cb86fb2b476cd066135bca7d8ef53678f35b53e7b72664548298c2cee98 WatchSource:0}: Error finding container 66ba9cb86fb2b476cd066135bca7d8ef53678f35b53e7b72664548298c2cee98: Status 404 returned error can't find the container with id 66ba9cb86fb2b476cd066135bca7d8ef53678f35b53e7b72664548298c2cee98 Oct 30 11:25:57.279863 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.279258 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z"] Oct 30 11:25:57.377783 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.377755 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" Oct 30 11:25:57.412338 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.412305 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2875bb2d-9cc3-4467-874a-bbacc4c94177-kube-api-access-2t9ts\" (UniqueName: \"kubernetes.io/projected/2875bb2d-9cc3-4467-874a-bbacc4c94177-kube-api-access-2t9ts\") pod \"2875bb2d-9cc3-4467-874a-bbacc4c94177\" (UID: \"2875bb2d-9cc3-4467-874a-bbacc4c94177\") " Oct 30 11:25:57.412540 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.412377 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2875bb2d-9cc3-4467-874a-bbacc4c94177-data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/2875bb2d-9cc3-4467-874a-bbacc4c94177-data-science-lokistack-querier-grpc\") pod \"2875bb2d-9cc3-4467-874a-bbacc4c94177\" (UID: \"2875bb2d-9cc3-4467-874a-bbacc4c94177\") " Oct 30 11:25:57.412540 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.412411 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2875bb2d-9cc3-4467-874a-bbacc4c94177-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2875bb2d-9cc3-4467-874a-bbacc4c94177-data-science-lokistack-ca-bundle\") pod \"2875bb2d-9cc3-4467-874a-bbacc4c94177\" (UID: \"2875bb2d-9cc3-4467-874a-bbacc4c94177\") " Oct 30 11:25:57.412540 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.412443 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2875bb2d-9cc3-4467-874a-bbacc4c94177-test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/2875bb2d-9cc3-4467-874a-bbacc4c94177-test-loki-shared-secret\") pod \"2875bb2d-9cc3-4467-874a-bbacc4c94177\" (UID: \"2875bb2d-9cc3-4467-874a-bbacc4c94177\") " Oct 30 11:25:57.412540 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.412484 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2875bb2d-9cc3-4467-874a-bbacc4c94177-data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/2875bb2d-9cc3-4467-874a-bbacc4c94177-data-science-lokistack-querier-http\") pod \"2875bb2d-9cc3-4467-874a-bbacc4c94177\" (UID: \"2875bb2d-9cc3-4467-874a-bbacc4c94177\") " Oct 30 11:25:57.412766 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.412580 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2875bb2d-9cc3-4467-874a-bbacc4c94177-config\" (UniqueName: \"kubernetes.io/configmap/2875bb2d-9cc3-4467-874a-bbacc4c94177-config\") pod \"2875bb2d-9cc3-4467-874a-bbacc4c94177\" (UID: \"2875bb2d-9cc3-4467-874a-bbacc4c94177\") " Oct 30 11:25:57.412885 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.412826 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2875bb2d-9cc3-4467-874a-bbacc4c94177-data-science-lokistack-ca-bundle" pod "2875bb2d-9cc3-4467-874a-bbacc4c94177" (UID: "2875bb2d-9cc3-4467-874a-bbacc4c94177"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:25:57.413246 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.413218 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2875bb2d-9cc3-4467-874a-bbacc4c94177-config" pod "2875bb2d-9cc3-4467-874a-bbacc4c94177" (UID: "2875bb2d-9cc3-4467-874a-bbacc4c94177"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:25:57.414699 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.414665 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2875bb2d-9cc3-4467-874a-bbacc4c94177-kube-api-access-2t9ts" pod "2875bb2d-9cc3-4467-874a-bbacc4c94177" (UID: "2875bb2d-9cc3-4467-874a-bbacc4c94177"). InnerVolumeSpecName "kube-api-access-2t9ts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:25:57.414981 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.414944 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2875bb2d-9cc3-4467-874a-bbacc4c94177-data-science-lokistack-querier-grpc" pod "2875bb2d-9cc3-4467-874a-bbacc4c94177" (UID: "2875bb2d-9cc3-4467-874a-bbacc4c94177"). InnerVolumeSpecName "data-science-lokistack-querier-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:25:57.415082 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.415041 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2875bb2d-9cc3-4467-874a-bbacc4c94177-data-science-lokistack-querier-http" pod "2875bb2d-9cc3-4467-874a-bbacc4c94177" (UID: "2875bb2d-9cc3-4467-874a-bbacc4c94177"). InnerVolumeSpecName "data-science-lokistack-querier-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:25:57.415271 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.415238 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2875bb2d-9cc3-4467-874a-bbacc4c94177-test-loki-shared-secret" pod "2875bb2d-9cc3-4467-874a-bbacc4c94177" (UID: "2875bb2d-9cc3-4467-874a-bbacc4c94177"). InnerVolumeSpecName "test-loki-shared-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:25:57.477062 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.477016 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" event={"ID":"11f6dbe1-349f-4ebb-b891-82e63db6d59a","Type":"ContainerStarted","Data":"2b3ff7d44994ddf9dcff5075165c7ba02110e9751a4529215408bdef92e5435f"} Oct 30 11:25:57.477062 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.477077 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" event={"ID":"11f6dbe1-349f-4ebb-b891-82e63db6d59a","Type":"ContainerStarted","Data":"66ba9cb86fb2b476cd066135bca7d8ef53678f35b53e7b72664548298c2cee98"} Oct 30 11:25:57.477753 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.477363 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" Oct 30 11:25:57.479782 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.479740 4289 generic.go:356] "Generic (PLEG): container finished" podID="2875bb2d-9cc3-4467-874a-bbacc4c94177" containerID="55498903d79bcfcd9b086eebd52b586ebf11c6a5b3109008afa5a2f7beff9dfe" exitCode=0 Oct 30 11:25:57.479782 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.479771 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" event={"ID":"2875bb2d-9cc3-4467-874a-bbacc4c94177","Type":"ContainerDied","Data":"55498903d79bcfcd9b086eebd52b586ebf11c6a5b3109008afa5a2f7beff9dfe"} Oct 30 11:25:57.479953 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.479806 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" event={"ID":"2875bb2d-9cc3-4467-874a-bbacc4c94177","Type":"ContainerDied","Data":"804099510b974ddada213f74278a5f70e77e9b1268901a94f99cccb69d2c7bec"} Oct 30 11:25:57.479953 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.479806 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g" Oct 30 11:25:57.479953 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.479831 4289 scope.go:122] "RemoveContainer" containerID="55498903d79bcfcd9b086eebd52b586ebf11c6a5b3109008afa5a2f7beff9dfe" Oct 30 11:25:57.491966 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.491945 4289 scope.go:122] "RemoveContainer" containerID="55498903d79bcfcd9b086eebd52b586ebf11c6a5b3109008afa5a2f7beff9dfe" Oct 30 11:25:57.492616 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:25:57.492572 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55498903d79bcfcd9b086eebd52b586ebf11c6a5b3109008afa5a2f7beff9dfe\": container with ID starting with 55498903d79bcfcd9b086eebd52b586ebf11c6a5b3109008afa5a2f7beff9dfe not found: ID does not exist" containerID="55498903d79bcfcd9b086eebd52b586ebf11c6a5b3109008afa5a2f7beff9dfe" Oct 30 11:25:57.492789 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.492742 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55498903d79bcfcd9b086eebd52b586ebf11c6a5b3109008afa5a2f7beff9dfe"} err="failed to get container status \"55498903d79bcfcd9b086eebd52b586ebf11c6a5b3109008afa5a2f7beff9dfe\": rpc error: code = NotFound desc = could not find container \"55498903d79bcfcd9b086eebd52b586ebf11c6a5b3109008afa5a2f7beff9dfe\": container with ID starting with 55498903d79bcfcd9b086eebd52b586ebf11c6a5b3109008afa5a2f7beff9dfe not found: ID does not exist" Oct 30 11:25:57.513729 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.513702 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2875bb2d-9cc3-4467-874a-bbacc4c94177-data-science-lokistack-ca-bundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:25:57.513729 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.513732 4289 reconciler_common.go:299] "Volume detached for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/2875bb2d-9cc3-4467-874a-bbacc4c94177-test-loki-shared-secret\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:25:57.513863 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.513743 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/2875bb2d-9cc3-4467-874a-bbacc4c94177-data-science-lokistack-querier-http\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:25:57.513863 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.513769 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2875bb2d-9cc3-4467-874a-bbacc4c94177-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:25:57.513863 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.513790 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t9ts\" (UniqueName: \"kubernetes.io/projected/2875bb2d-9cc3-4467-874a-bbacc4c94177-kube-api-access-2t9ts\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:25:57.513863 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.513807 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/2875bb2d-9cc3-4467-874a-bbacc4c94177-data-science-lokistack-querier-grpc\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:25:57.520575 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.520534 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" podStartSLOduration=1.520523726 podStartE2EDuration="1.520523726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:25:57.519893204 +0000 UTC m=+2448.021272689" watchObservedRunningTime="2026-10-30 11:25:57.520523726 +0000 UTC m=+2448.021903211" Oct 30 11:25:57.554808 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.554755 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g"] Oct 30 11:25:57.556943 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.556924 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-84985db556-4vt4g"] Oct 30 11:25:57.624415 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:57.624390 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2875bb2d-9cc3-4467-874a-bbacc4c94177" path="/var/lib/kubelet/pods/2875bb2d-9cc3-4467-874a-bbacc4c94177/volumes" Oct 30 11:25:58.157419 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:25:58.157338 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:26:05.659694 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:05.659659 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 11:26:05.660210 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:05.659911 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" podUID="11866315-a378-46ca-a34a-da39ef3a8e51" containerName="loki-index-gateway" containerID="cri-o://f8f8cccc51cbe4e873ebe8d9718db06b9d7ed73357bacb2af466031d12dd95ee" gracePeriod=30 Oct 30 11:26:05.934731 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:05.934707 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:05.990643 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:05.990611 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11866315-a378-46ca-a34a-da39ef3a8e51-data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/11866315-a378-46ca-a34a-da39ef3a8e51-data-science-lokistack-index-gateway-grpc\") pod \"11866315-a378-46ca-a34a-da39ef3a8e51\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " Oct 30 11:26:05.990822 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:05.990663 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11866315-a378-46ca-a34a-da39ef3a8e51-test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/11866315-a378-46ca-a34a-da39ef3a8e51-test-loki-shared-secret\") pod \"11866315-a378-46ca-a34a-da39ef3a8e51\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " Oct 30 11:26:05.990822 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:05.990696 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11866315-a378-46ca-a34a-da39ef3a8e51-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11866315-a378-46ca-a34a-da39ef3a8e51-data-science-lokistack-ca-bundle\") pod \"11866315-a378-46ca-a34a-da39ef3a8e51\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " Oct 30 11:26:05.990822 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:05.990744 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11866315-a378-46ca-a34a-da39ef3a8e51-config\" (UniqueName: \"kubernetes.io/configmap/11866315-a378-46ca-a34a-da39ef3a8e51-config\") pod \"11866315-a378-46ca-a34a-da39ef3a8e51\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " Oct 30 11:26:05.990822 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:05.990820 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") pod \"11866315-a378-46ca-a34a-da39ef3a8e51\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " Oct 30 11:26:05.991047 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:05.990855 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11866315-a378-46ca-a34a-da39ef3a8e51-data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/11866315-a378-46ca-a34a-da39ef3a8e51-data-science-lokistack-index-gateway-http\") pod \"11866315-a378-46ca-a34a-da39ef3a8e51\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " Oct 30 11:26:05.991047 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:05.990889 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11866315-a378-46ca-a34a-da39ef3a8e51-kube-api-access-tbg8f\" (UniqueName: \"kubernetes.io/projected/11866315-a378-46ca-a34a-da39ef3a8e51-kube-api-access-tbg8f\") pod \"11866315-a378-46ca-a34a-da39ef3a8e51\" (UID: \"11866315-a378-46ca-a34a-da39ef3a8e51\") " Oct 30 11:26:05.991164 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:05.991142 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11866315-a378-46ca-a34a-da39ef3a8e51-data-science-lokistack-ca-bundle" pod "11866315-a378-46ca-a34a-da39ef3a8e51" (UID: "11866315-a378-46ca-a34a-da39ef3a8e51"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:26:05.991229 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:05.991201 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11866315-a378-46ca-a34a-da39ef3a8e51-config" pod "11866315-a378-46ca-a34a-da39ef3a8e51" (UID: "11866315-a378-46ca-a34a-da39ef3a8e51"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:26:05.993276 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:05.993250 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11866315-a378-46ca-a34a-da39ef3a8e51-data-science-lokistack-index-gateway-http" pod "11866315-a378-46ca-a34a-da39ef3a8e51" (UID: "11866315-a378-46ca-a34a-da39ef3a8e51"). InnerVolumeSpecName "data-science-lokistack-index-gateway-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:26:05.993421 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:05.993278 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11866315-a378-46ca-a34a-da39ef3a8e51-test-loki-shared-secret" pod "11866315-a378-46ca-a34a-da39ef3a8e51" (UID: "11866315-a378-46ca-a34a-da39ef3a8e51"). InnerVolumeSpecName "test-loki-shared-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:26:05.993559 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:05.993538 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11866315-a378-46ca-a34a-da39ef3a8e51-kube-api-access-tbg8f" pod "11866315-a378-46ca-a34a-da39ef3a8e51" (UID: "11866315-a378-46ca-a34a-da39ef3a8e51"). InnerVolumeSpecName "kube-api-access-tbg8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:26:05.993607 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:05.993598 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11866315-a378-46ca-a34a-da39ef3a8e51-data-science-lokistack-index-gateway-grpc" pod "11866315-a378-46ca-a34a-da39ef3a8e51" (UID: "11866315-a378-46ca-a34a-da39ef3a8e51"). InnerVolumeSpecName "data-science-lokistack-index-gateway-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:26:05.994329 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:05.994308 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3" pod "11866315-a378-46ca-a34a-da39ef3a8e51" (UID: "11866315-a378-46ca-a34a-da39ef3a8e51"). InnerVolumeSpecName "pvc-c30998fa-a6d9-4900-9b47-20d22faf37db". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 11:26:06.091792 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.091766 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/11866315-a378-46ca-a34a-da39ef3a8e51-data-science-lokistack-index-gateway-grpc\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:26:06.091792 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.091791 4289 reconciler_common.go:299] "Volume detached for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/11866315-a378-46ca-a34a-da39ef3a8e51-test-loki-shared-secret\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:26:06.091995 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.091803 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11866315-a378-46ca-a34a-da39ef3a8e51-data-science-lokistack-ca-bundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:26:06.091995 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.091813 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11866315-a378-46ca-a34a-da39ef3a8e51-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:26:06.091995 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.091851 4289 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") on node \"ip-10-0-137-242.ec2.internal\" " Oct 30 11:26:06.091995 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.091864 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/11866315-a378-46ca-a34a-da39ef3a8e51-data-science-lokistack-index-gateway-http\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:26:06.091995 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.091873 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbg8f\" (UniqueName: \"kubernetes.io/projected/11866315-a378-46ca-a34a-da39ef3a8e51-kube-api-access-tbg8f\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:26:06.103509 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.103490 4289 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-c30998fa-a6d9-4900-9b47-20d22faf37db" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3") on node "ip-10-0-137-242.ec2.internal" Oct 30 11:26:06.192979 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.192906 4289 reconciler_common.go:299] "Volume detached for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"csi-2e669b6d2bbdfa3a6b46036bf22d4a7a6aba8888c2c130325f5aeb55a396d7c5\"" Oct 30 11:26:06.520328 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.520291 4289 generic.go:356] "Generic (PLEG): container finished" podID="11866315-a378-46ca-a34a-da39ef3a8e51" containerID="f8f8cccc51cbe4e873ebe8d9718db06b9d7ed73357bacb2af466031d12dd95ee" exitCode=0 Oct 30 11:26:06.520515 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.520342 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:06.520515 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.520371 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"11866315-a378-46ca-a34a-da39ef3a8e51","Type":"ContainerDied","Data":"f8f8cccc51cbe4e873ebe8d9718db06b9d7ed73357bacb2af466031d12dd95ee"} Oct 30 11:26:06.520515 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.520462 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"11866315-a378-46ca-a34a-da39ef3a8e51","Type":"ContainerDied","Data":"6d9fc55074054900b0238143edcb8e920b9586af5b80e00ae27520073b84140b"} Oct 30 11:26:06.520515 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.520484 4289 scope.go:122] "RemoveContainer" containerID="f8f8cccc51cbe4e873ebe8d9718db06b9d7ed73357bacb2af466031d12dd95ee" Oct 30 11:26:06.529326 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.529307 4289 scope.go:122] "RemoveContainer" containerID="f8f8cccc51cbe4e873ebe8d9718db06b9d7ed73357bacb2af466031d12dd95ee" Oct 30 11:26:06.529599 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:26:06.529580 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f8cccc51cbe4e873ebe8d9718db06b9d7ed73357bacb2af466031d12dd95ee\": container with ID starting with f8f8cccc51cbe4e873ebe8d9718db06b9d7ed73357bacb2af466031d12dd95ee not found: ID does not exist" containerID="f8f8cccc51cbe4e873ebe8d9718db06b9d7ed73357bacb2af466031d12dd95ee" Oct 30 11:26:06.529674 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.529612 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f8cccc51cbe4e873ebe8d9718db06b9d7ed73357bacb2af466031d12dd95ee"} err="failed to get container status \"f8f8cccc51cbe4e873ebe8d9718db06b9d7ed73357bacb2af466031d12dd95ee\": rpc error: code = NotFound desc = could not find container \"f8f8cccc51cbe4e873ebe8d9718db06b9d7ed73357bacb2af466031d12dd95ee\": container with ID starting with f8f8cccc51cbe4e873ebe8d9718db06b9d7ed73357bacb2af466031d12dd95ee not found: ID does not exist" Oct 30 11:26:06.574390 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.574359 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 11:26:06.586021 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.585997 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 11:26:06.617943 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.617919 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 11:26:06.618252 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.618240 4289 cpu_manager.go:414] "containerMap: removing container" podUID="2875bb2d-9cc3-4467-874a-bbacc4c94177" containerName="loki-querier" Oct 30 11:26:06.618300 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.618253 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2875bb2d-9cc3-4467-874a-bbacc4c94177" containerName="loki-querier" Oct 30 11:26:06.618300 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.618264 4289 cpu_manager.go:414] "containerMap: removing container" podUID="11866315-a378-46ca-a34a-da39ef3a8e51" containerName="loki-index-gateway" Oct 30 11:26:06.618300 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.618269 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11866315-a378-46ca-a34a-da39ef3a8e51" containerName="loki-index-gateway" Oct 30 11:26:06.618390 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.618324 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="2875bb2d-9cc3-4467-874a-bbacc4c94177" containerName="loki-querier" Oct 30 11:26:06.618390 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.618332 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="11866315-a378-46ca-a34a-da39ef3a8e51" containerName="loki-index-gateway" Oct 30 11:26:06.622748 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.622727 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:06.637404 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.637387 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-grpc\"" Oct 30 11:26:06.637514 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.637498 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-http\"" Oct 30 11:26:06.642398 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.642378 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 11:26:06.696485 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.696456 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/013b474e-89fc-4473-bf9e-e832bbd8b4a4-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:06.696817 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.696499 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/013b474e-89fc-4473-bf9e-e832bbd8b4a4-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:06.696817 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.696526 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/013b474e-89fc-4473-bf9e-e832bbd8b4a4-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:06.696817 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.696603 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/013b474e-89fc-4473-bf9e-e832bbd8b4a4-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:06.696817 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.696669 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrqp8\" (UniqueName: \"kubernetes.io/projected/013b474e-89fc-4473-bf9e-e832bbd8b4a4-kube-api-access-lrqp8\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:06.696817 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.696811 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/013b474e-89fc-4473-bf9e-e832bbd8b4a4-test-loki-lifecycle-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:06.797912 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.797831 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/013b474e-89fc-4473-bf9e-e832bbd8b4a4-test-loki-lifecycle-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:06.797912 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.797889 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/013b474e-89fc-4473-bf9e-e832bbd8b4a4-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:06.798129 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.797915 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/013b474e-89fc-4473-bf9e-e832bbd8b4a4-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:06.798129 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.797937 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/013b474e-89fc-4473-bf9e-e832bbd8b4a4-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:06.798129 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.797958 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/013b474e-89fc-4473-bf9e-e832bbd8b4a4-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:06.798284 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.798126 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrqp8\" (UniqueName: \"kubernetes.io/projected/013b474e-89fc-4473-bf9e-e832bbd8b4a4-kube-api-access-lrqp8\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:06.798834 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.798806 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/013b474e-89fc-4473-bf9e-e832bbd8b4a4-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:06.798965 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.798858 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/013b474e-89fc-4473-bf9e-e832bbd8b4a4-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:06.800477 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.800425 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/013b474e-89fc-4473-bf9e-e832bbd8b4a4-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:06.800638 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.800614 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/013b474e-89fc-4473-bf9e-e832bbd8b4a4-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:06.800689 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.800621 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/013b474e-89fc-4473-bf9e-e832bbd8b4a4-test-loki-lifecycle-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:06.824209 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:06.824180 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrqp8\" (UniqueName: \"kubernetes.io/projected/013b474e-89fc-4473-bf9e-e832bbd8b4a4-kube-api-access-lrqp8\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:07.623863 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:07.623830 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11866315-a378-46ca-a34a-da39ef3a8e51" path="/var/lib/kubelet/pods/11866315-a378-46ca-a34a-da39ef3a8e51/volumes" Oct 30 11:26:15.578590 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:15.578549 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:15.607289 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:15.607259 4289 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:15.679522 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:15.679492 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:15.679678 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:15.679598 4289 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:15.694024 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:15.694002 4289 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") DevicePath \"csi-2e669b6d2bbdfa3a6b46036bf22d4a7a6aba8888c2c130325f5aeb55a396d7c5\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:15.797575 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:15.797544 4289 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b48470aea0a0a5b9ac05ab05099d84e77041b5209a87fa2cabb2ba39e3c57c0c/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:15.823625 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:15.823597 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:15.933959 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:15.933871 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:16.230721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:16.230691 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 11:26:16.231401 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:26:16.231375 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod013b474e_89fc_4473_bf9e_e832bbd8b4a4.slice/crio-12a63d26383020f3076daf4b81ca63b3bb28a0dc7ca49e0d27eb7a93276716de WatchSource:0}: Error finding container 12a63d26383020f3076daf4b81ca63b3bb28a0dc7ca49e0d27eb7a93276716de: Status 404 returned error can't find the container with id 12a63d26383020f3076daf4b81ca63b3bb28a0dc7ca49e0d27eb7a93276716de Oct 30 11:26:16.564774 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:16.564678 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"013b474e-89fc-4473-bf9e-e832bbd8b4a4","Type":"ContainerStarted","Data":"7739a61fdcad68ae07e462a9e5b8a7012b90f3eae2ae63e58543da190ee6f357"} Oct 30 11:26:16.564774 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:16.564736 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"013b474e-89fc-4473-bf9e-e832bbd8b4a4","Type":"ContainerStarted","Data":"12a63d26383020f3076daf4b81ca63b3bb28a0dc7ca49e0d27eb7a93276716de"} Oct 30 11:26:16.564774 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:16.564763 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:16.606859 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:16.606814 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" podStartSLOduration=10.606801038 podStartE2EDuration="10.606801038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:26:16.606496706 +0000 UTC m=+2467.107876191" watchObservedRunningTime="2026-10-30 11:26:16.606801038 +0000 UTC m=+2467.108180524" Oct 30 11:26:18.490830 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:18.490799 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" Oct 30 11:26:25.366210 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:25.366120 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"] Oct 30 11:26:25.366736 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:25.366333 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podUID="5976f8fd-b82f-492c-93e2-f0b01f0d9e60" containerName="loki-ingester" containerID="cri-o://5648fa4c64daf07346c3856878e2da64cc2a2712e42dfb882d5dbcea9df63cba" gracePeriod=600 Oct 30 11:26:26.257133 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:26:26.257094 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:26:26.412123 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.412099 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:26.579165 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.579126 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-data-science-lokistack-ingester-grpc\") pod \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " Oct 30 11:26:26.579347 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.579182 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-data-science-lokistack-ca-bundle\") pod \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " Oct 30 11:26:26.579347 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.579234 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-kube-api-access-mzjwk\" (UniqueName: \"kubernetes.io/projected/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-kube-api-access-mzjwk\") pod \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " Oct 30 11:26:26.579347 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.579257 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-data-science-lokistack-ingester-http\") pod \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " Oct 30 11:26:26.579347 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.579343 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973\") pod \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " Oct 30 11:26:26.579598 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.579394 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-config\" (UniqueName: \"kubernetes.io/configmap/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-config\") pod \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " Oct 30 11:26:26.579598 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.579449 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-test-loki-shared-secret\") pod \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " Oct 30 11:26:26.579598 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.579526 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3\") pod \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\" (UID: \"5976f8fd-b82f-492c-93e2-f0b01f0d9e60\") " Oct 30 11:26:26.579949 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.579751 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-data-science-lokistack-ca-bundle" pod "5976f8fd-b82f-492c-93e2-f0b01f0d9e60" (UID: "5976f8fd-b82f-492c-93e2-f0b01f0d9e60"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:26:26.579949 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.579848 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-data-science-lokistack-ca-bundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:26:26.579949 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.579888 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-config" pod "5976f8fd-b82f-492c-93e2-f0b01f0d9e60" (UID: "5976f8fd-b82f-492c-93e2-f0b01f0d9e60"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:26:26.581737 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.581708 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-data-science-lokistack-ingester-http" pod "5976f8fd-b82f-492c-93e2-f0b01f0d9e60" (UID: "5976f8fd-b82f-492c-93e2-f0b01f0d9e60"). InnerVolumeSpecName "data-science-lokistack-ingester-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:26:26.581909 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.581883 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-data-science-lokistack-ingester-grpc" pod "5976f8fd-b82f-492c-93e2-f0b01f0d9e60" (UID: "5976f8fd-b82f-492c-93e2-f0b01f0d9e60"). InnerVolumeSpecName "data-science-lokistack-ingester-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:26:26.582151 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.582121 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-kube-api-access-mzjwk" pod "5976f8fd-b82f-492c-93e2-f0b01f0d9e60" (UID: "5976f8fd-b82f-492c-93e2-f0b01f0d9e60"). InnerVolumeSpecName "kube-api-access-mzjwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:26:26.582231 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.582186 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-test-loki-shared-secret" pod "5976f8fd-b82f-492c-93e2-f0b01f0d9e60" (UID: "5976f8fd-b82f-492c-93e2-f0b01f0d9e60"). InnerVolumeSpecName "test-loki-shared-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:26:26.582550 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.582527 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973" pod "5976f8fd-b82f-492c-93e2-f0b01f0d9e60" (UID: "5976f8fd-b82f-492c-93e2-f0b01f0d9e60"). InnerVolumeSpecName "pvc-6dcdb7ad-d1ff-4871-aad9-1c2bebcc5bb8". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 11:26:26.582634 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.582547 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3" pod "5976f8fd-b82f-492c-93e2-f0b01f0d9e60" (UID: "5976f8fd-b82f-492c-93e2-f0b01f0d9e60"). InnerVolumeSpecName "pvc-5016d491-1af7-4119-a668-95a1697a696b". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 11:26:26.606998 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.606971 4289 generic.go:356] "Generic (PLEG): container finished" podID="5976f8fd-b82f-492c-93e2-f0b01f0d9e60" containerID="5648fa4c64daf07346c3856878e2da64cc2a2712e42dfb882d5dbcea9df63cba" exitCode=0 Oct 30 11:26:26.607083 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.607025 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:26.607083 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.607052 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" event={"ID":"5976f8fd-b82f-492c-93e2-f0b01f0d9e60","Type":"ContainerDied","Data":"5648fa4c64daf07346c3856878e2da64cc2a2712e42dfb882d5dbcea9df63cba"} Oct 30 11:26:26.607166 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.607102 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" event={"ID":"5976f8fd-b82f-492c-93e2-f0b01f0d9e60","Type":"ContainerDied","Data":"d69921d0c5ae80c18c8db6a5ec15ab4a318db17c32c1db10ff0e29be6b819764"} Oct 30 11:26:26.607166 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.607123 4289 scope.go:122] "RemoveContainer" containerID="5648fa4c64daf07346c3856878e2da64cc2a2712e42dfb882d5dbcea9df63cba" Oct 30 11:26:26.615809 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.615790 4289 scope.go:122] "RemoveContainer" containerID="5648fa4c64daf07346c3856878e2da64cc2a2712e42dfb882d5dbcea9df63cba" Oct 30 11:26:26.616052 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:26:26.616035 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5648fa4c64daf07346c3856878e2da64cc2a2712e42dfb882d5dbcea9df63cba\": container with ID starting with 5648fa4c64daf07346c3856878e2da64cc2a2712e42dfb882d5dbcea9df63cba not found: ID does not exist" containerID="5648fa4c64daf07346c3856878e2da64cc2a2712e42dfb882d5dbcea9df63cba" Oct 30 11:26:26.616098 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.616059 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5648fa4c64daf07346c3856878e2da64cc2a2712e42dfb882d5dbcea9df63cba"} err="failed to get container status \"5648fa4c64daf07346c3856878e2da64cc2a2712e42dfb882d5dbcea9df63cba\": rpc error: code = NotFound desc = could not find container \"5648fa4c64daf07346c3856878e2da64cc2a2712e42dfb882d5dbcea9df63cba\": container with ID starting with 5648fa4c64daf07346c3856878e2da64cc2a2712e42dfb882d5dbcea9df63cba not found: ID does not exist" Oct 30 11:26:26.656389 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.656364 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"] Oct 30 11:26:26.661141 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.661118 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"] Oct 30 11:26:26.680881 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.680858 4289 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6dcdb7ad-d1ff-4871-aad9-1c2bebcc5bb8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973\") on node \"ip-10-0-137-242.ec2.internal\" " Oct 30 11:26:26.680881 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.680880 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:26:26.680990 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.680891 4289 reconciler_common.go:299] "Volume detached for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-test-loki-shared-secret\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:26:26.680990 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.680907 4289 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-5016d491-1af7-4119-a668-95a1697a696b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3\") on node \"ip-10-0-137-242.ec2.internal\" " Oct 30 11:26:26.680990 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.680923 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-data-science-lokistack-ingester-grpc\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:26:26.680990 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.680937 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzjwk\" (UniqueName: \"kubernetes.io/projected/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-kube-api-access-mzjwk\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:26:26.680990 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.680949 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/5976f8fd-b82f-492c-93e2-f0b01f0d9e60-data-science-lokistack-ingester-http\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:26:26.693932 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.693914 4289 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-5016d491-1af7-4119-a668-95a1697a696b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3") on node "ip-10-0-137-242.ec2.internal" Oct 30 11:26:26.694152 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.694140 4289 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6dcdb7ad-d1ff-4871-aad9-1c2bebcc5bb8" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973") on node "ip-10-0-137-242.ec2.internal" Oct 30 11:26:26.699353 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.699328 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"] Oct 30 11:26:26.699741 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.699725 4289 cpu_manager.go:414] "containerMap: removing container" podUID="5976f8fd-b82f-492c-93e2-f0b01f0d9e60" containerName="loki-ingester" Oct 30 11:26:26.699813 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.699743 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5976f8fd-b82f-492c-93e2-f0b01f0d9e60" containerName="loki-ingester" Oct 30 11:26:26.699883 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.699820 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="5976f8fd-b82f-492c-93e2-f0b01f0d9e60" containerName="loki-ingester" Oct 30 11:26:26.704103 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.704083 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:26.718581 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.718563 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-grpc\"" Oct 30 11:26:26.718671 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.718656 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-http\"" Oct 30 11:26:26.724524 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.724503 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"] Oct 30 11:26:26.781495 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.781461 4289 reconciler_common.go:299] "Volume detached for volume \"pvc-5016d491-1af7-4119-a668-95a1697a696b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"csi-24162340a6be733c56063f2faeaf5eac3653e9c6e6136a764bd2aa7236d709ba\"" Oct 30 11:26:26.781495 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.781491 4289 reconciler_common.go:299] "Volume detached for volume \"pvc-6dcdb7ad-d1ff-4871-aad9-1c2bebcc5bb8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"csi-fed6c8c359d982cd671ddd7b8904c0992b96b06853cad388882421bf5eda486c\"" Oct 30 11:26:26.882081 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.882006 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5edc29a7-fe63-41b1-993c-712b0351b0a8-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:26.882081 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.882039 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5edc29a7-fe63-41b1-993c-712b0351b0a8-config\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:26.882243 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.882121 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb7lp\" (UniqueName: \"kubernetes.io/projected/5edc29a7-fe63-41b1-993c-712b0351b0a8-kube-api-access-sb7lp\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:26.882243 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.882172 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/5edc29a7-fe63-41b1-993c-712b0351b0a8-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:26.882243 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.882198 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/5edc29a7-fe63-41b1-993c-712b0351b0a8-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:26.882346 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.882256 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/5edc29a7-fe63-41b1-993c-712b0351b0a8-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:26.982783 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.982746 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/5edc29a7-fe63-41b1-993c-712b0351b0a8-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:26.982783 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.982787 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5edc29a7-fe63-41b1-993c-712b0351b0a8-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:26.983016 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.982809 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5edc29a7-fe63-41b1-993c-712b0351b0a8-config\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:26.983016 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.982867 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb7lp\" (UniqueName: \"kubernetes.io/projected/5edc29a7-fe63-41b1-993c-712b0351b0a8-kube-api-access-sb7lp\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:26.983016 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.982908 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/5edc29a7-fe63-41b1-993c-712b0351b0a8-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:26.983016 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.982937 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/5edc29a7-fe63-41b1-993c-712b0351b0a8-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:26.983495 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.983465 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5edc29a7-fe63-41b1-993c-712b0351b0a8-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:26.983670 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.983641 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5edc29a7-fe63-41b1-993c-712b0351b0a8-config\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:26.985356 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.985331 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/5edc29a7-fe63-41b1-993c-712b0351b0a8-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:26.985446 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.985411 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/5edc29a7-fe63-41b1-993c-712b0351b0a8-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:26.985593 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:26.985572 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/5edc29a7-fe63-41b1-993c-712b0351b0a8-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:27.005847 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:27.005825 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb7lp\" (UniqueName: \"kubernetes.io/projected/5edc29a7-fe63-41b1-993c-712b0351b0a8-kube-api-access-sb7lp\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:27.624994 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:27.624958 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5976f8fd-b82f-492c-93e2-f0b01f0d9e60" path="/var/lib/kubelet/pods/5976f8fd-b82f-492c-93e2-f0b01f0d9e60/volumes" Oct 30 11:26:35.856255 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:35.856220 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-5016d491-1af7-4119-a668-95a1697a696b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:35.856716 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:35.856285 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6dcdb7ad-d1ff-4871-aad9-1c2bebcc5bb8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:35.896492 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:35.896459 4289 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6dcdb7ad-d1ff-4871-aad9-1c2bebcc5bb8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:35.896492 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:35.896459 4289 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-5016d491-1af7-4119-a668-95a1697a696b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:35.956880 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:35.956850 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-5016d491-1af7-4119-a668-95a1697a696b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:35.957032 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:35.956896 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6dcdb7ad-d1ff-4871-aad9-1c2bebcc5bb8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:35.957032 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:35.956972 4289 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6dcdb7ad-d1ff-4871-aad9-1c2bebcc5bb8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:35.957032 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:35.956985 4289 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-5016d491-1af7-4119-a668-95a1697a696b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:35.971191 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:35.971162 4289 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6dcdb7ad-d1ff-4871-aad9-1c2bebcc5bb8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") DevicePath \"csi-fed6c8c359d982cd671ddd7b8904c0992b96b06853cad388882421bf5eda486c\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:35.971304 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:35.971220 4289 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-5016d491-1af7-4119-a668-95a1697a696b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") DevicePath \"csi-24162340a6be733c56063f2faeaf5eac3653e9c6e6136a764bd2aa7236d709ba\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:36.075395 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:36.075363 4289 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6dcdb7ad-d1ff-4871-aad9-1c2bebcc5bb8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/186c84c83760be5837750810e291c0bfee785cbd5ee4b0582f60daafa9018035/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:36.083307 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:36.083276 4289 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-5016d491-1af7-4119-a668-95a1697a696b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/0d90e1e43de51125033a7f82cd0645765bf678ea2057dc8845fad0d0b4c35b0f/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:36.106940 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:36.106884 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-5016d491-1af7-4119-a668-95a1697a696b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:36.108368 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:36.108347 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6dcdb7ad-d1ff-4871-aad9-1c2bebcc5bb8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973\") pod \"data-science-lokistack-ingester-1\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:36.315344 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:36.315311 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:36.625310 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:36.625282 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"] Oct 30 11:26:36.625792 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:26:36.625769 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5edc29a7_fe63_41b1_993c_712b0351b0a8.slice/crio-dacf189cb74068131adb9969af2ca50d8294c6c9e42b9e812dea6670216c3cba WatchSource:0}: Error finding container dacf189cb74068131adb9969af2ca50d8294c6c9e42b9e812dea6670216c3cba: Status 404 returned error can't find the container with id dacf189cb74068131adb9969af2ca50d8294c6c9e42b9e812dea6670216c3cba Oct 30 11:26:36.650868 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:36.650842 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" event={"ID":"5edc29a7-fe63-41b1-993c-712b0351b0a8","Type":"ContainerStarted","Data":"dacf189cb74068131adb9969af2ca50d8294c6c9e42b9e812dea6670216c3cba"} Oct 30 11:26:37.573908 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:37.573883 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:26:37.657485 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:37.657450 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" event={"ID":"5edc29a7-fe63-41b1-993c-712b0351b0a8","Type":"ContainerStarted","Data":"da560c733e83031c3386f250c13f01e5ed44ca7670dfb67e527497e4d67896fd"} Oct 30 11:26:37.657659 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:37.657575 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:26:37.701206 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:37.701158 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podStartSLOduration=11.701144591 podStartE2EDuration="11.701144591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:26:37.699807685 +0000 UTC m=+2488.201187174" watchObservedRunningTime="2026-10-30 11:26:37.701144591 +0000 UTC m=+2488.202524076" Oct 30 11:26:56.258301 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:26:56.258267 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:26:58.270250 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:58.270209 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:26:58.667779 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:26:58.667692 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podUID="5edc29a7-fe63-41b1-993c-712b0351b0a8" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 11:27:08.665853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:27:08.665815 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podUID="5edc29a7-fe63-41b1-993c-712b0351b0a8" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 11:27:18.666116 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:27:18.666085 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:27:26.257647 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:27:26.257613 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:27:56.257653 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:27:56.257564 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:27:58.324772 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:27:58.324723 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:28:16.731400 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:16.731361 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:28:16.917383 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:16.917342 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:28:17.084561 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:17.084469 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:28:18.938805 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:18.938416 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:28:18.966145 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:18.966112 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 11:28:18.966349 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:18.966325 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" podUID="013b474e-89fc-4473-bf9e-e832bbd8b4a4" containerName="loki-index-gateway" containerID="cri-o://7739a61fdcad68ae07e462a9e5b8a7012b90f3eae2ae63e58543da190ee6f357" gracePeriod=30 Oct 30 11:28:19.051105 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.051073 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"] Oct 30 11:28:19.051354 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.051299 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podUID="5edc29a7-fe63-41b1-993c-712b0351b0a8" containerName="loki-ingester" containerID="cri-o://da560c733e83031c3386f250c13f01e5ed44ca7670dfb67e527497e4d67896fd" gracePeriod=600 Oct 30 11:28:19.067880 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.067850 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj"] Oct 30 11:28:19.068089 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.068067 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" podUID="551b0f59-4902-49a5-a83f-85bad32dee9e" containerName="gateway" containerID="cri-o://c64a94339ad91ba61a9f913009603b94e1b43e788c3e533297445587da582c6c" gracePeriod=30 Oct 30 11:28:19.068184 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.068100 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" podUID="551b0f59-4902-49a5-a83f-85bad32dee9e" containerName="opa" containerID="cri-o://308a55367339f15b085527624420d6c6479a31b22e52a52de4d0274c0bc51c91" gracePeriod=30 Oct 30 11:28:19.082847 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.082818 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z"] Oct 30 11:28:19.083067 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.083044 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" podUID="11f6dbe1-349f-4ebb-b891-82e63db6d59a" containerName="loki-querier" containerID="cri-o://2b3ff7d44994ddf9dcff5075165c7ba02110e9751a4529215408bdef92e5435f" gracePeriod=30 Oct 30 11:28:19.096991 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.096960 4289 generic.go:356] "Generic (PLEG): container finished" podID="013b474e-89fc-4473-bf9e-e832bbd8b4a4" containerID="7739a61fdcad68ae07e462a9e5b8a7012b90f3eae2ae63e58543da190ee6f357" exitCode=0 Oct 30 11:28:19.097117 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.097024 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"013b474e-89fc-4473-bf9e-e832bbd8b4a4","Type":"ContainerDied","Data":"7739a61fdcad68ae07e462a9e5b8a7012b90f3eae2ae63e58543da190ee6f357"} Oct 30 11:28:19.097186 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.097154 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x"] Oct 30 11:28:19.097377 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.097353 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" podUID="a517cbd2-329c-4c01-8f6f-955220592158" containerName="loki-distributor" containerID="cri-o://1fb3677247ef00a9b02ecebc668da95486722d4219c5592fe4180c0da05d8bcf" gracePeriod=30 Oct 30 11:28:19.155239 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.154941 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:28:19.280333 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.280296 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:19.325484 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.325451 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:28:19.338329 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.338305 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/013b474e-89fc-4473-bf9e-e832bbd8b4a4-data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/013b474e-89fc-4473-bf9e-e832bbd8b4a4-data-science-lokistack-index-gateway-grpc\") pod \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " Oct 30 11:28:19.338470 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.338342 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/013b474e-89fc-4473-bf9e-e832bbd8b4a4-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/013b474e-89fc-4473-bf9e-e832bbd8b4a4-data-science-lokistack-ca-bundle\") pod \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " Oct 30 11:28:19.338470 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.338381 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/013b474e-89fc-4473-bf9e-e832bbd8b4a4-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/013b474e-89fc-4473-bf9e-e832bbd8b4a4-test-loki-lifecycle-secret\") pod \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " Oct 30 11:28:19.338596 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.338473 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") pod \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " Oct 30 11:28:19.338596 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.338503 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/013b474e-89fc-4473-bf9e-e832bbd8b4a4-config\" (UniqueName: \"kubernetes.io/configmap/013b474e-89fc-4473-bf9e-e832bbd8b4a4-config\") pod \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " Oct 30 11:28:19.338596 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.338535 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/013b474e-89fc-4473-bf9e-e832bbd8b4a4-kube-api-access-lrqp8\" (UniqueName: \"kubernetes.io/projected/013b474e-89fc-4473-bf9e-e832bbd8b4a4-kube-api-access-lrqp8\") pod \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " Oct 30 11:28:19.338596 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.338578 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/013b474e-89fc-4473-bf9e-e832bbd8b4a4-data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/013b474e-89fc-4473-bf9e-e832bbd8b4a4-data-science-lokistack-index-gateway-http\") pod \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\" (UID: \"013b474e-89fc-4473-bf9e-e832bbd8b4a4\") " Oct 30 11:28:19.338862 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.338790 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/013b474e-89fc-4473-bf9e-e832bbd8b4a4-data-science-lokistack-ca-bundle" pod "013b474e-89fc-4473-bf9e-e832bbd8b4a4" (UID: "013b474e-89fc-4473-bf9e-e832bbd8b4a4"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:28:19.339136 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.339110 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/013b474e-89fc-4473-bf9e-e832bbd8b4a4-config" pod "013b474e-89fc-4473-bf9e-e832bbd8b4a4" (UID: "013b474e-89fc-4473-bf9e-e832bbd8b4a4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:28:19.339251 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.339163 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/013b474e-89fc-4473-bf9e-e832bbd8b4a4-data-science-lokistack-ca-bundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:19.341570 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.341548 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013b474e-89fc-4473-bf9e-e832bbd8b4a4-kube-api-access-lrqp8" pod "013b474e-89fc-4473-bf9e-e832bbd8b4a4" (UID: "013b474e-89fc-4473-bf9e-e832bbd8b4a4"). InnerVolumeSpecName "kube-api-access-lrqp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:28:19.341684 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.341588 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/013b474e-89fc-4473-bf9e-e832bbd8b4a4-data-science-lokistack-index-gateway-http" pod "013b474e-89fc-4473-bf9e-e832bbd8b4a4" (UID: "013b474e-89fc-4473-bf9e-e832bbd8b4a4"). InnerVolumeSpecName "data-science-lokistack-index-gateway-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:28:19.342111 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.342091 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3" pod "013b474e-89fc-4473-bf9e-e832bbd8b4a4" (UID: "013b474e-89fc-4473-bf9e-e832bbd8b4a4"). InnerVolumeSpecName "pvc-c30998fa-a6d9-4900-9b47-20d22faf37db". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 11:28:19.345552 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.345527 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/013b474e-89fc-4473-bf9e-e832bbd8b4a4-test-loki-lifecycle-secret" pod "013b474e-89fc-4473-bf9e-e832bbd8b4a4" (UID: "013b474e-89fc-4473-bf9e-e832bbd8b4a4"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:28:19.347221 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.347202 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/013b474e-89fc-4473-bf9e-e832bbd8b4a4-data-science-lokistack-index-gateway-grpc" pod "013b474e-89fc-4473-bf9e-e832bbd8b4a4" (UID: "013b474e-89fc-4473-bf9e-e832bbd8b4a4"). InnerVolumeSpecName "data-science-lokistack-index-gateway-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:28:19.440553 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.440522 4289 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/013b474e-89fc-4473-bf9e-e832bbd8b4a4-test-loki-lifecycle-secret\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:19.440701 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.440573 4289 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") on node \"ip-10-0-137-242.ec2.internal\" " Oct 30 11:28:19.440701 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.440588 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/013b474e-89fc-4473-bf9e-e832bbd8b4a4-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:19.440701 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.440601 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrqp8\" (UniqueName: \"kubernetes.io/projected/013b474e-89fc-4473-bf9e-e832bbd8b4a4-kube-api-access-lrqp8\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:19.440701 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.440615 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/013b474e-89fc-4473-bf9e-e832bbd8b4a4-data-science-lokistack-index-gateway-http\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:19.440701 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.440629 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/013b474e-89fc-4473-bf9e-e832bbd8b4a4-data-science-lokistack-index-gateway-grpc\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:19.450951 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.450933 4289 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-c30998fa-a6d9-4900-9b47-20d22faf37db" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3") on node "ip-10-0-137-242.ec2.internal" Oct 30 11:28:19.541525 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.541424 4289 reconciler_common.go:299] "Volume detached for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"csi-2e669b6d2bbdfa3a6b46036bf22d4a7a6aba8888c2c130325f5aeb55a396d7c5\"" Oct 30 11:28:19.670243 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.670216 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" Oct 30 11:28:19.743037 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.743003 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11f6dbe1-349f-4ebb-b891-82e63db6d59a-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11f6dbe1-349f-4ebb-b891-82e63db6d59a-data-science-lokistack-ca-bundle\") pod \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\" (UID: \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\") " Oct 30 11:28:19.743199 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.743064 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f6dbe1-349f-4ebb-b891-82e63db6d59a-kube-api-access-7v6jk\" (UniqueName: \"kubernetes.io/projected/11f6dbe1-349f-4ebb-b891-82e63db6d59a-kube-api-access-7v6jk\") pod \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\" (UID: \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\") " Oct 30 11:28:19.743199 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.743085 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11f6dbe1-349f-4ebb-b891-82e63db6d59a-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/11f6dbe1-349f-4ebb-b891-82e63db6d59a-test-loki-lifecycle-secret\") pod \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\" (UID: \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\") " Oct 30 11:28:19.743199 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.743111 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11f6dbe1-349f-4ebb-b891-82e63db6d59a-data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/11f6dbe1-349f-4ebb-b891-82e63db6d59a-data-science-lokistack-querier-http\") pod \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\" (UID: \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\") " Oct 30 11:28:19.743199 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.743150 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11f6dbe1-349f-4ebb-b891-82e63db6d59a-config\" (UniqueName: \"kubernetes.io/configmap/11f6dbe1-349f-4ebb-b891-82e63db6d59a-config\") pod \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\" (UID: \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\") " Oct 30 11:28:19.743396 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.743201 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11f6dbe1-349f-4ebb-b891-82e63db6d59a-data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/11f6dbe1-349f-4ebb-b891-82e63db6d59a-data-science-lokistack-querier-grpc\") pod \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\" (UID: \"11f6dbe1-349f-4ebb-b891-82e63db6d59a\") " Oct 30 11:28:19.743612 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.743464 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f6dbe1-349f-4ebb-b891-82e63db6d59a-data-science-lokistack-ca-bundle" pod "11f6dbe1-349f-4ebb-b891-82e63db6d59a" (UID: "11f6dbe1-349f-4ebb-b891-82e63db6d59a"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:28:19.743747 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.743675 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f6dbe1-349f-4ebb-b891-82e63db6d59a-config" pod "11f6dbe1-349f-4ebb-b891-82e63db6d59a" (UID: "11f6dbe1-349f-4ebb-b891-82e63db6d59a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:28:19.745372 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.745337 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f6dbe1-349f-4ebb-b891-82e63db6d59a-data-science-lokistack-querier-grpc" pod "11f6dbe1-349f-4ebb-b891-82e63db6d59a" (UID: "11f6dbe1-349f-4ebb-b891-82e63db6d59a"). InnerVolumeSpecName "data-science-lokistack-querier-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:28:19.745499 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.745401 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f6dbe1-349f-4ebb-b891-82e63db6d59a-data-science-lokistack-querier-http" pod "11f6dbe1-349f-4ebb-b891-82e63db6d59a" (UID: "11f6dbe1-349f-4ebb-b891-82e63db6d59a"). InnerVolumeSpecName "data-science-lokistack-querier-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:28:19.745499 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.745421 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f6dbe1-349f-4ebb-b891-82e63db6d59a-kube-api-access-7v6jk" pod "11f6dbe1-349f-4ebb-b891-82e63db6d59a" (UID: "11f6dbe1-349f-4ebb-b891-82e63db6d59a"). InnerVolumeSpecName "kube-api-access-7v6jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:28:19.745499 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.745420 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f6dbe1-349f-4ebb-b891-82e63db6d59a-test-loki-lifecycle-secret" pod "11f6dbe1-349f-4ebb-b891-82e63db6d59a" (UID: "11f6dbe1-349f-4ebb-b891-82e63db6d59a"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:28:19.844805 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.844778 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11f6dbe1-349f-4ebb-b891-82e63db6d59a-data-science-lokistack-ca-bundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:19.844805 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.844802 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v6jk\" (UniqueName: \"kubernetes.io/projected/11f6dbe1-349f-4ebb-b891-82e63db6d59a-kube-api-access-7v6jk\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:19.844956 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.844815 4289 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/11f6dbe1-349f-4ebb-b891-82e63db6d59a-test-loki-lifecycle-secret\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:19.844956 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.844824 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/11f6dbe1-349f-4ebb-b891-82e63db6d59a-data-science-lokistack-querier-http\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:19.844956 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.844834 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11f6dbe1-349f-4ebb-b891-82e63db6d59a-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:19.844956 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:19.844843 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/11f6dbe1-349f-4ebb-b891-82e63db6d59a-data-science-lokistack-querier-grpc\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:20.003547 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.003523 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:28:20.047370 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.047347 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3\") pod \"5edc29a7-fe63-41b1-993c-712b0351b0a8\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " Oct 30 11:28:20.047544 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.047507 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5edc29a7-fe63-41b1-993c-712b0351b0a8-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5edc29a7-fe63-41b1-993c-712b0351b0a8-data-science-lokistack-ca-bundle\") pod \"5edc29a7-fe63-41b1-993c-712b0351b0a8\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " Oct 30 11:28:20.047705 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.047572 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5edc29a7-fe63-41b1-993c-712b0351b0a8-kube-api-access-sb7lp\" (UniqueName: \"kubernetes.io/projected/5edc29a7-fe63-41b1-993c-712b0351b0a8-kube-api-access-sb7lp\") pod \"5edc29a7-fe63-41b1-993c-712b0351b0a8\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " Oct 30 11:28:20.047705 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.047612 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5edc29a7-fe63-41b1-993c-712b0351b0a8-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/5edc29a7-fe63-41b1-993c-712b0351b0a8-test-loki-lifecycle-secret\") pod \"5edc29a7-fe63-41b1-993c-712b0351b0a8\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " Oct 30 11:28:20.047905 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.047793 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5edc29a7-fe63-41b1-993c-712b0351b0a8-data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/5edc29a7-fe63-41b1-993c-712b0351b0a8-data-science-lokistack-ingester-http\") pod \"5edc29a7-fe63-41b1-993c-712b0351b0a8\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " Oct 30 11:28:20.047905 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.047891 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5edc29a7-fe63-41b1-993c-712b0351b0a8-data-science-lokistack-ca-bundle" pod "5edc29a7-fe63-41b1-993c-712b0351b0a8" (UID: "5edc29a7-fe63-41b1-993c-712b0351b0a8"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:28:20.048545 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.048523 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973\") pod \"5edc29a7-fe63-41b1-993c-712b0351b0a8\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " Oct 30 11:28:20.049217 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.049192 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5edc29a7-fe63-41b1-993c-712b0351b0a8-config\" (UniqueName: \"kubernetes.io/configmap/5edc29a7-fe63-41b1-993c-712b0351b0a8-config\") pod \"5edc29a7-fe63-41b1-993c-712b0351b0a8\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " Oct 30 11:28:20.049625 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.049580 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5edc29a7-fe63-41b1-993c-712b0351b0a8-config" pod "5edc29a7-fe63-41b1-993c-712b0351b0a8" (UID: "5edc29a7-fe63-41b1-993c-712b0351b0a8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:28:20.049799 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.049781 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5edc29a7-fe63-41b1-993c-712b0351b0a8-data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/5edc29a7-fe63-41b1-993c-712b0351b0a8-data-science-lokistack-ingester-grpc\") pod \"5edc29a7-fe63-41b1-993c-712b0351b0a8\" (UID: \"5edc29a7-fe63-41b1-993c-712b0351b0a8\") " Oct 30 11:28:20.050569 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.050549 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5edc29a7-fe63-41b1-993c-712b0351b0a8-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:20.050896 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.050808 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5edc29a7-fe63-41b1-993c-712b0351b0a8-data-science-lokistack-ca-bundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:20.051022 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.050631 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5edc29a7-fe63-41b1-993c-712b0351b0a8-data-science-lokistack-ingester-http" pod "5edc29a7-fe63-41b1-993c-712b0351b0a8" (UID: "5edc29a7-fe63-41b1-993c-712b0351b0a8"). InnerVolumeSpecName "data-science-lokistack-ingester-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:28:20.051101 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.050657 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5edc29a7-fe63-41b1-993c-712b0351b0a8-kube-api-access-sb7lp" pod "5edc29a7-fe63-41b1-993c-712b0351b0a8" (UID: "5edc29a7-fe63-41b1-993c-712b0351b0a8"). InnerVolumeSpecName "kube-api-access-sb7lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:28:20.051167 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.051130 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5edc29a7-fe63-41b1-993c-712b0351b0a8-test-loki-lifecycle-secret" pod "5edc29a7-fe63-41b1-993c-712b0351b0a8" (UID: "5edc29a7-fe63-41b1-993c-712b0351b0a8"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:28:20.051229 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.051176 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3" pod "5edc29a7-fe63-41b1-993c-712b0351b0a8" (UID: "5edc29a7-fe63-41b1-993c-712b0351b0a8"). InnerVolumeSpecName "pvc-5016d491-1af7-4119-a668-95a1697a696b". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 11:28:20.052071 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.052050 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5edc29a7-fe63-41b1-993c-712b0351b0a8-data-science-lokistack-ingester-grpc" pod "5edc29a7-fe63-41b1-993c-712b0351b0a8" (UID: "5edc29a7-fe63-41b1-993c-712b0351b0a8"). InnerVolumeSpecName "data-science-lokistack-ingester-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:28:20.052772 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.052752 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973" pod "5edc29a7-fe63-41b1-993c-712b0351b0a8" (UID: "5edc29a7-fe63-41b1-993c-712b0351b0a8"). InnerVolumeSpecName "pvc-6dcdb7ad-d1ff-4871-aad9-1c2bebcc5bb8". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 11:28:20.103024 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.102991 4289 generic.go:356] "Generic (PLEG): container finished" podID="11f6dbe1-349f-4ebb-b891-82e63db6d59a" containerID="2b3ff7d44994ddf9dcff5075165c7ba02110e9751a4529215408bdef92e5435f" exitCode=0 Oct 30 11:28:20.103146 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.103044 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" Oct 30 11:28:20.103146 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.103079 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" event={"ID":"11f6dbe1-349f-4ebb-b891-82e63db6d59a","Type":"ContainerDied","Data":"2b3ff7d44994ddf9dcff5075165c7ba02110e9751a4529215408bdef92e5435f"} Oct 30 11:28:20.103146 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.103137 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z" event={"ID":"11f6dbe1-349f-4ebb-b891-82e63db6d59a","Type":"ContainerDied","Data":"66ba9cb86fb2b476cd066135bca7d8ef53678f35b53e7b72664548298c2cee98"} Oct 30 11:28:20.103287 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.103166 4289 scope.go:122] "RemoveContainer" containerID="2b3ff7d44994ddf9dcff5075165c7ba02110e9751a4529215408bdef92e5435f" Oct 30 11:28:20.104502 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.104480 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:20.104646 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.104471 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"013b474e-89fc-4473-bf9e-e832bbd8b4a4","Type":"ContainerDied","Data":"12a63d26383020f3076daf4b81ca63b3bb28a0dc7ca49e0d27eb7a93276716de"} Oct 30 11:28:20.106736 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.106707 4289 generic.go:356] "Generic (PLEG): container finished" podID="551b0f59-4902-49a5-a83f-85bad32dee9e" containerID="308a55367339f15b085527624420d6c6479a31b22e52a52de4d0274c0bc51c91" exitCode=0 Oct 30 11:28:20.106885 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.106786 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" event={"ID":"551b0f59-4902-49a5-a83f-85bad32dee9e","Type":"ContainerDied","Data":"308a55367339f15b085527624420d6c6479a31b22e52a52de4d0274c0bc51c91"} Oct 30 11:28:20.108249 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.108224 4289 generic.go:356] "Generic (PLEG): container finished" podID="5edc29a7-fe63-41b1-993c-712b0351b0a8" containerID="da560c733e83031c3386f250c13f01e5ed44ca7670dfb67e527497e4d67896fd" exitCode=0 Oct 30 11:28:20.108337 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.108266 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 11:28:20.108337 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.108298 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" event={"ID":"5edc29a7-fe63-41b1-993c-712b0351b0a8","Type":"ContainerDied","Data":"da560c733e83031c3386f250c13f01e5ed44ca7670dfb67e527497e4d67896fd"} Oct 30 11:28:20.108408 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.108339 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" event={"ID":"5edc29a7-fe63-41b1-993c-712b0351b0a8","Type":"ContainerDied","Data":"dacf189cb74068131adb9969af2ca50d8294c6c9e42b9e812dea6670216c3cba"} Oct 30 11:28:20.113565 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.113528 4289 scope.go:122] "RemoveContainer" containerID="2b3ff7d44994ddf9dcff5075165c7ba02110e9751a4529215408bdef92e5435f" Oct 30 11:28:20.113828 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:28:20.113801 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b3ff7d44994ddf9dcff5075165c7ba02110e9751a4529215408bdef92e5435f\": container with ID starting with 2b3ff7d44994ddf9dcff5075165c7ba02110e9751a4529215408bdef92e5435f not found: ID does not exist" containerID="2b3ff7d44994ddf9dcff5075165c7ba02110e9751a4529215408bdef92e5435f" Oct 30 11:28:20.113923 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.113835 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3ff7d44994ddf9dcff5075165c7ba02110e9751a4529215408bdef92e5435f"} err="failed to get container status \"2b3ff7d44994ddf9dcff5075165c7ba02110e9751a4529215408bdef92e5435f\": rpc error: code = NotFound desc = could not find container \"2b3ff7d44994ddf9dcff5075165c7ba02110e9751a4529215408bdef92e5435f\": container with ID starting with 2b3ff7d44994ddf9dcff5075165c7ba02110e9751a4529215408bdef92e5435f not found: ID does not exist" Oct 30 11:28:20.113923 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.113857 4289 scope.go:122] "RemoveContainer" containerID="7739a61fdcad68ae07e462a9e5b8a7012b90f3eae2ae63e58543da190ee6f357" Oct 30 11:28:20.122513 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.122497 4289 scope.go:122] "RemoveContainer" containerID="da560c733e83031c3386f250c13f01e5ed44ca7670dfb67e527497e4d67896fd" Oct 30 11:28:20.132541 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.132524 4289 scope.go:122] "RemoveContainer" containerID="da560c733e83031c3386f250c13f01e5ed44ca7670dfb67e527497e4d67896fd" Oct 30 11:28:20.132819 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:28:20.132795 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da560c733e83031c3386f250c13f01e5ed44ca7670dfb67e527497e4d67896fd\": container with ID starting with da560c733e83031c3386f250c13f01e5ed44ca7670dfb67e527497e4d67896fd not found: ID does not exist" containerID="da560c733e83031c3386f250c13f01e5ed44ca7670dfb67e527497e4d67896fd" Oct 30 11:28:20.132904 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.132820 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da560c733e83031c3386f250c13f01e5ed44ca7670dfb67e527497e4d67896fd"} err="failed to get container status \"da560c733e83031c3386f250c13f01e5ed44ca7670dfb67e527497e4d67896fd\": rpc error: code = NotFound desc = could not find container \"da560c733e83031c3386f250c13f01e5ed44ca7670dfb67e527497e4d67896fd\": container with ID starting with da560c733e83031c3386f250c13f01e5ed44ca7670dfb67e527497e4d67896fd not found: ID does not exist" Oct 30 11:28:20.151880 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.151859 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/5edc29a7-fe63-41b1-993c-712b0351b0a8-data-science-lokistack-ingester-grpc\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:20.151960 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.151905 4289 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-5016d491-1af7-4119-a668-95a1697a696b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3\") on node \"ip-10-0-137-242.ec2.internal\" " Oct 30 11:28:20.151960 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.151920 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb7lp\" (UniqueName: \"kubernetes.io/projected/5edc29a7-fe63-41b1-993c-712b0351b0a8-kube-api-access-sb7lp\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:20.151960 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.151930 4289 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/5edc29a7-fe63-41b1-993c-712b0351b0a8-test-loki-lifecycle-secret\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:20.151960 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.151938 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/5edc29a7-fe63-41b1-993c-712b0351b0a8-data-science-lokistack-ingester-http\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:20.151960 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.151952 4289 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6dcdb7ad-d1ff-4871-aad9-1c2bebcc5bb8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973\") on node \"ip-10-0-137-242.ec2.internal\" " Oct 30 11:28:20.153323 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.153302 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"] Oct 30 11:28:20.157156 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.157137 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"] Oct 30 11:28:20.164415 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.164401 4289 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-5016d491-1af7-4119-a668-95a1697a696b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3") on node "ip-10-0-137-242.ec2.internal" Oct 30 11:28:20.167522 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.167504 4289 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6dcdb7ad-d1ff-4871-aad9-1c2bebcc5bb8" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973") on node "ip-10-0-137-242.ec2.internal" Oct 30 11:28:20.189936 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.189909 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 11:28:20.196210 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.196188 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 11:28:20.228835 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.228802 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z"] Oct 30 11:28:20.235169 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.235144 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-6kc8z"] Oct 30 11:28:20.252599 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.252577 4289 reconciler_common.go:299] "Volume detached for volume \"pvc-6dcdb7ad-d1ff-4871-aad9-1c2bebcc5bb8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00cd6a945cbbaf973\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"csi-fed6c8c359d982cd671ddd7b8904c0992b96b06853cad388882421bf5eda486c\"" Oct 30 11:28:20.252599 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.252600 4289 reconciler_common.go:299] "Volume detached for volume \"pvc-5016d491-1af7-4119-a668-95a1697a696b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00202e19da6748af3\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"csi-24162340a6be733c56063f2faeaf5eac3653e9c6e6136a764bd2aa7236d709ba\"" Oct 30 11:28:20.322438 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.322404 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:28:20.353068 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.353035 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/551b0f59-4902-49a5-a83f-85bad32dee9e-data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/551b0f59-4902-49a5-a83f-85bad32dee9e-data-science-lokistack-gateway-client-http\") pod \"551b0f59-4902-49a5-a83f-85bad32dee9e\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " Oct 30 11:28:20.353247 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.353095 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-tls-secret\" (UniqueName: \"kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-tls-secret\") pod \"551b0f59-4902-49a5-a83f-85bad32dee9e\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " Oct 30 11:28:20.353247 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.353126 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-data-science-lokistack-ca-bundle\") pod \"551b0f59-4902-49a5-a83f-85bad32dee9e\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " Oct 30 11:28:20.353247 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.353163 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-lokistack-gateway\") pod \"551b0f59-4902-49a5-a83f-85bad32dee9e\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " Oct 30 11:28:20.353247 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.353207 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-rbac\" (UniqueName: \"kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-rbac\") pod \"551b0f59-4902-49a5-a83f-85bad32dee9e\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " Oct 30 11:28:20.353247 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.353245 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-data-science-lokistack-gateway-ca-bundle\") pod \"551b0f59-4902-49a5-a83f-85bad32dee9e\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " Oct 30 11:28:20.353552 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.353279 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-kube-api-access-ss76v\" (UniqueName: \"kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-kube-api-access-ss76v\") pod \"551b0f59-4902-49a5-a83f-85bad32dee9e\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " Oct 30 11:28:20.353552 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.353382 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/551b0f59-4902-49a5-a83f-85bad32dee9e-tenants\" (UniqueName: \"kubernetes.io/secret/551b0f59-4902-49a5-a83f-85bad32dee9e-tenants\") pod \"551b0f59-4902-49a5-a83f-85bad32dee9e\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " Oct 30 11:28:20.353552 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.353520 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-data-science-lokistack-ca-bundle" pod "551b0f59-4902-49a5-a83f-85bad32dee9e" (UID: "551b0f59-4902-49a5-a83f-85bad32dee9e"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:28:20.353768 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.353743 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-data-science-lokistack-ca-bundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:20.355914 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.355870 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/551b0f59-4902-49a5-a83f-85bad32dee9e-data-science-lokistack-gateway-client-http" pod "551b0f59-4902-49a5-a83f-85bad32dee9e" (UID: "551b0f59-4902-49a5-a83f-85bad32dee9e"). InnerVolumeSpecName "data-science-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:28:20.356221 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.356195 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-tls-secret" pod "551b0f59-4902-49a5-a83f-85bad32dee9e" (UID: "551b0f59-4902-49a5-a83f-85bad32dee9e"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:28:20.358169 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.358145 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-data-science-lokistack-gateway-ca-bundle" pod "551b0f59-4902-49a5-a83f-85bad32dee9e" (UID: "551b0f59-4902-49a5-a83f-85bad32dee9e"). InnerVolumeSpecName "data-science-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:28:20.359061 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:28:20.359038 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-rbac podName:551b0f59-4902-49a5-a83f-85bad32dee9e nodeName:}" failed. No retries permitted until 2026-10-30 11:28:20.85901342 +0000 UTC m=+2591.360392903 (durationBeforeRetry 500ms). Error: error cleaning subPath mounts for volume "kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-rbac" (UniqueName: "kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-rbac") pod "551b0f59-4902-49a5-a83f-85bad32dee9e" (UID: "551b0f59-4902-49a5-a83f-85bad32dee9e") : error deleting /var/lib/kubelet/pods/551b0f59-4902-49a5-a83f-85bad32dee9e/volume-subpaths: remove /var/lib/kubelet/pods/551b0f59-4902-49a5-a83f-85bad32dee9e/volume-subpaths: no such file or directory Oct 30 11:28:20.359270 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.359251 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-lokistack-gateway" pod "551b0f59-4902-49a5-a83f-85bad32dee9e" (UID: "551b0f59-4902-49a5-a83f-85bad32dee9e"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:28:20.359544 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.359525 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-kube-api-access-ss76v" pod "551b0f59-4902-49a5-a83f-85bad32dee9e" (UID: "551b0f59-4902-49a5-a83f-85bad32dee9e"). InnerVolumeSpecName "kube-api-access-ss76v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:28:20.360482 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.360409 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/551b0f59-4902-49a5-a83f-85bad32dee9e-tenants" pod "551b0f59-4902-49a5-a83f-85bad32dee9e" (UID: "551b0f59-4902-49a5-a83f-85bad32dee9e"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:28:20.447955 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.447933 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" Oct 30 11:28:20.454982 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.454965 4289 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-tls-secret\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:20.455040 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.454984 4289 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-lokistack-gateway\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:20.455040 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.454998 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-data-science-lokistack-gateway-ca-bundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:20.455040 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.455011 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss76v\" (UniqueName: \"kubernetes.io/projected/551b0f59-4902-49a5-a83f-85bad32dee9e-kube-api-access-ss76v\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:20.455040 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.455020 4289 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/551b0f59-4902-49a5-a83f-85bad32dee9e-tenants\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:20.455040 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.455029 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/551b0f59-4902-49a5-a83f-85bad32dee9e-data-science-lokistack-gateway-client-http\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:20.575724 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.575094 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a517cbd2-329c-4c01-8f6f-955220592158-data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/a517cbd2-329c-4c01-8f6f-955220592158-data-science-lokistack-distributor-http\") pod \"a517cbd2-329c-4c01-8f6f-955220592158\" (UID: \"a517cbd2-329c-4c01-8f6f-955220592158\") " Oct 30 11:28:20.575724 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.575184 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a517cbd2-329c-4c01-8f6f-955220592158-config\" (UniqueName: \"kubernetes.io/configmap/a517cbd2-329c-4c01-8f6f-955220592158-config\") pod \"a517cbd2-329c-4c01-8f6f-955220592158\" (UID: \"a517cbd2-329c-4c01-8f6f-955220592158\") " Oct 30 11:28:20.575724 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.575229 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a517cbd2-329c-4c01-8f6f-955220592158-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a517cbd2-329c-4c01-8f6f-955220592158-data-science-lokistack-ca-bundle\") pod \"a517cbd2-329c-4c01-8f6f-955220592158\" (UID: \"a517cbd2-329c-4c01-8f6f-955220592158\") " Oct 30 11:28:20.575724 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.575267 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a517cbd2-329c-4c01-8f6f-955220592158-data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/a517cbd2-329c-4c01-8f6f-955220592158-data-science-lokistack-distributor-grpc\") pod \"a517cbd2-329c-4c01-8f6f-955220592158\" (UID: \"a517cbd2-329c-4c01-8f6f-955220592158\") " Oct 30 11:28:20.575724 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.575310 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a517cbd2-329c-4c01-8f6f-955220592158-kube-api-access-2wzd5\" (UniqueName: \"kubernetes.io/projected/a517cbd2-329c-4c01-8f6f-955220592158-kube-api-access-2wzd5\") pod \"a517cbd2-329c-4c01-8f6f-955220592158\" (UID: \"a517cbd2-329c-4c01-8f6f-955220592158\") " Oct 30 11:28:20.576605 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.576377 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a517cbd2-329c-4c01-8f6f-955220592158-data-science-lokistack-ca-bundle" pod "a517cbd2-329c-4c01-8f6f-955220592158" (UID: "a517cbd2-329c-4c01-8f6f-955220592158"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:28:20.576605 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.576564 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a517cbd2-329c-4c01-8f6f-955220592158-config" pod "a517cbd2-329c-4c01-8f6f-955220592158" (UID: "a517cbd2-329c-4c01-8f6f-955220592158"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:28:20.578536 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.578491 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a517cbd2-329c-4c01-8f6f-955220592158-data-science-lokistack-distributor-http" pod "a517cbd2-329c-4c01-8f6f-955220592158" (UID: "a517cbd2-329c-4c01-8f6f-955220592158"). InnerVolumeSpecName "data-science-lokistack-distributor-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:28:20.578657 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.578633 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a517cbd2-329c-4c01-8f6f-955220592158-kube-api-access-2wzd5" pod "a517cbd2-329c-4c01-8f6f-955220592158" (UID: "a517cbd2-329c-4c01-8f6f-955220592158"). InnerVolumeSpecName "kube-api-access-2wzd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:28:20.578823 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.578806 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a517cbd2-329c-4c01-8f6f-955220592158-data-science-lokistack-distributor-grpc" pod "a517cbd2-329c-4c01-8f6f-955220592158" (UID: "a517cbd2-329c-4c01-8f6f-955220592158"). InnerVolumeSpecName "data-science-lokistack-distributor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:28:20.676694 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.676671 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a517cbd2-329c-4c01-8f6f-955220592158-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:20.676694 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.676695 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a517cbd2-329c-4c01-8f6f-955220592158-data-science-lokistack-ca-bundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:20.676946 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.676705 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/a517cbd2-329c-4c01-8f6f-955220592158-data-science-lokistack-distributor-grpc\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:20.676946 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.676718 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wzd5\" (UniqueName: \"kubernetes.io/projected/a517cbd2-329c-4c01-8f6f-955220592158-kube-api-access-2wzd5\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:20.676946 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.676727 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/a517cbd2-329c-4c01-8f6f-955220592158-data-science-lokistack-distributor-http\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:20.878303 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.878212 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-rbac\" (UniqueName: \"kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-rbac\") pod \"551b0f59-4902-49a5-a83f-85bad32dee9e\" (UID: \"551b0f59-4902-49a5-a83f-85bad32dee9e\") " Oct 30 11:28:20.878549 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.878528 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-rbac" pod "551b0f59-4902-49a5-a83f-85bad32dee9e" (UID: "551b0f59-4902-49a5-a83f-85bad32dee9e"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:28:20.978873 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:20.978844 4289 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/551b0f59-4902-49a5-a83f-85bad32dee9e-rbac\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:28:21.115061 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.115027 4289 generic.go:356] "Generic (PLEG): container finished" podID="a517cbd2-329c-4c01-8f6f-955220592158" containerID="1fb3677247ef00a9b02ecebc668da95486722d4219c5592fe4180c0da05d8bcf" exitCode=0 Oct 30 11:28:21.115511 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.115072 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" Oct 30 11:28:21.115511 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.115101 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" event={"ID":"a517cbd2-329c-4c01-8f6f-955220592158","Type":"ContainerDied","Data":"1fb3677247ef00a9b02ecebc668da95486722d4219c5592fe4180c0da05d8bcf"} Oct 30 11:28:21.115511 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.115148 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x" event={"ID":"a517cbd2-329c-4c01-8f6f-955220592158","Type":"ContainerDied","Data":"ce2b3dbc00322ae4e9ab5d16435dffeb13f020e2c417a8234b69250b07ac75c5"} Oct 30 11:28:21.115511 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.115178 4289 scope.go:122] "RemoveContainer" containerID="1fb3677247ef00a9b02ecebc668da95486722d4219c5592fe4180c0da05d8bcf" Oct 30 11:28:21.117838 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.117811 4289 generic.go:356] "Generic (PLEG): container finished" podID="551b0f59-4902-49a5-a83f-85bad32dee9e" containerID="c64a94339ad91ba61a9f913009603b94e1b43e788c3e533297445587da582c6c" exitCode=0 Oct 30 11:28:21.117964 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.117856 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" Oct 30 11:28:21.117964 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.117891 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" event={"ID":"551b0f59-4902-49a5-a83f-85bad32dee9e","Type":"ContainerDied","Data":"c64a94339ad91ba61a9f913009603b94e1b43e788c3e533297445587da582c6c"} Oct 30 11:28:21.117964 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.117937 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj" event={"ID":"551b0f59-4902-49a5-a83f-85bad32dee9e","Type":"ContainerDied","Data":"7ca97f1eb43afa54a74e8896793dce135cd0f6f783282cf3900221b0716dba8e"} Oct 30 11:28:21.124003 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.123842 4289 scope.go:122] "RemoveContainer" containerID="1fb3677247ef00a9b02ecebc668da95486722d4219c5592fe4180c0da05d8bcf" Oct 30 11:28:21.124311 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:28:21.124267 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fb3677247ef00a9b02ecebc668da95486722d4219c5592fe4180c0da05d8bcf\": container with ID starting with 1fb3677247ef00a9b02ecebc668da95486722d4219c5592fe4180c0da05d8bcf not found: ID does not exist" containerID="1fb3677247ef00a9b02ecebc668da95486722d4219c5592fe4180c0da05d8bcf" Oct 30 11:28:21.124395 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.124315 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fb3677247ef00a9b02ecebc668da95486722d4219c5592fe4180c0da05d8bcf"} err="failed to get container status \"1fb3677247ef00a9b02ecebc668da95486722d4219c5592fe4180c0da05d8bcf\": rpc error: code = NotFound desc = could not find container \"1fb3677247ef00a9b02ecebc668da95486722d4219c5592fe4180c0da05d8bcf\": container with ID starting with 1fb3677247ef00a9b02ecebc668da95486722d4219c5592fe4180c0da05d8bcf not found: ID does not exist" Oct 30 11:28:21.124395 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.124331 4289 scope.go:122] "RemoveContainer" containerID="308a55367339f15b085527624420d6c6479a31b22e52a52de4d0274c0bc51c91" Oct 30 11:28:21.131989 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.131971 4289 scope.go:122] "RemoveContainer" containerID="c64a94339ad91ba61a9f913009603b94e1b43e788c3e533297445587da582c6c" Oct 30 11:28:21.139225 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.139205 4289 scope.go:122] "RemoveContainer" containerID="308a55367339f15b085527624420d6c6479a31b22e52a52de4d0274c0bc51c91" Oct 30 11:28:21.139535 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:28:21.139514 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"308a55367339f15b085527624420d6c6479a31b22e52a52de4d0274c0bc51c91\": container with ID starting with 308a55367339f15b085527624420d6c6479a31b22e52a52de4d0274c0bc51c91 not found: ID does not exist" containerID="308a55367339f15b085527624420d6c6479a31b22e52a52de4d0274c0bc51c91" Oct 30 11:28:21.139623 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.139546 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308a55367339f15b085527624420d6c6479a31b22e52a52de4d0274c0bc51c91"} err="failed to get container status \"308a55367339f15b085527624420d6c6479a31b22e52a52de4d0274c0bc51c91\": rpc error: code = NotFound desc = could not find container \"308a55367339f15b085527624420d6c6479a31b22e52a52de4d0274c0bc51c91\": container with ID starting with 308a55367339f15b085527624420d6c6479a31b22e52a52de4d0274c0bc51c91 not found: ID does not exist" Oct 30 11:28:21.139623 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.139568 4289 scope.go:122] "RemoveContainer" containerID="c64a94339ad91ba61a9f913009603b94e1b43e788c3e533297445587da582c6c" Oct 30 11:28:21.139862 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:28:21.139826 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c64a94339ad91ba61a9f913009603b94e1b43e788c3e533297445587da582c6c\": container with ID starting with c64a94339ad91ba61a9f913009603b94e1b43e788c3e533297445587da582c6c not found: ID does not exist" containerID="c64a94339ad91ba61a9f913009603b94e1b43e788c3e533297445587da582c6c" Oct 30 11:28:21.139910 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.139872 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c64a94339ad91ba61a9f913009603b94e1b43e788c3e533297445587da582c6c"} err="failed to get container status \"c64a94339ad91ba61a9f913009603b94e1b43e788c3e533297445587da582c6c\": rpc error: code = NotFound desc = could not find container \"c64a94339ad91ba61a9f913009603b94e1b43e788c3e533297445587da582c6c\": container with ID starting with c64a94339ad91ba61a9f913009603b94e1b43e788c3e533297445587da582c6c not found: ID does not exist" Oct 30 11:28:21.165107 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.165087 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj"] Oct 30 11:28:21.176618 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.176590 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-698f8cbf65-btsfj"] Oct 30 11:28:21.211053 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.205614 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x"] Oct 30 11:28:21.211053 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.210592 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-jtn4x"] Oct 30 11:28:21.623770 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.623741 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="013b474e-89fc-4473-bf9e-e832bbd8b4a4" path="/var/lib/kubelet/pods/013b474e-89fc-4473-bf9e-e832bbd8b4a4/volumes" Oct 30 11:28:21.624151 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.624138 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f6dbe1-349f-4ebb-b891-82e63db6d59a" path="/var/lib/kubelet/pods/11f6dbe1-349f-4ebb-b891-82e63db6d59a/volumes" Oct 30 11:28:21.624523 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.624510 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="551b0f59-4902-49a5-a83f-85bad32dee9e" path="/var/lib/kubelet/pods/551b0f59-4902-49a5-a83f-85bad32dee9e/volumes" Oct 30 11:28:21.625249 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.625235 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5edc29a7-fe63-41b1-993c-712b0351b0a8" path="/var/lib/kubelet/pods/5edc29a7-fe63-41b1-993c-712b0351b0a8/volumes" Oct 30 11:28:21.625607 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:21.625594 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a517cbd2-329c-4c01-8f6f-955220592158" path="/var/lib/kubelet/pods/a517cbd2-329c-4c01-8f6f-955220592158/volumes" Oct 30 11:28:23.584402 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.584364 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf"] Oct 30 11:28:23.584892 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.584830 4289 cpu_manager.go:414] "containerMap: removing container" podUID="551b0f59-4902-49a5-a83f-85bad32dee9e" containerName="gateway" Oct 30 11:28:23.584892 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.584847 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551b0f59-4902-49a5-a83f-85bad32dee9e" containerName="gateway" Oct 30 11:28:23.584892 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.584863 4289 cpu_manager.go:414] "containerMap: removing container" podUID="11f6dbe1-349f-4ebb-b891-82e63db6d59a" containerName="loki-querier" Oct 30 11:28:23.584892 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.584871 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f6dbe1-349f-4ebb-b891-82e63db6d59a" containerName="loki-querier" Oct 30 11:28:23.585119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.584909 4289 cpu_manager.go:414] "containerMap: removing container" podUID="551b0f59-4902-49a5-a83f-85bad32dee9e" containerName="opa" Oct 30 11:28:23.585119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.584917 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551b0f59-4902-49a5-a83f-85bad32dee9e" containerName="opa" Oct 30 11:28:23.585119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.584932 4289 cpu_manager.go:414] "containerMap: removing container" podUID="013b474e-89fc-4473-bf9e-e832bbd8b4a4" containerName="loki-index-gateway" Oct 30 11:28:23.585119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.584939 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="013b474e-89fc-4473-bf9e-e832bbd8b4a4" containerName="loki-index-gateway" Oct 30 11:28:23.585119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.584955 4289 cpu_manager.go:414] "containerMap: removing container" podUID="a517cbd2-329c-4c01-8f6f-955220592158" containerName="loki-distributor" Oct 30 11:28:23.585119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.584962 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a517cbd2-329c-4c01-8f6f-955220592158" containerName="loki-distributor" Oct 30 11:28:23.585119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.584978 4289 cpu_manager.go:414] "containerMap: removing container" podUID="5edc29a7-fe63-41b1-993c-712b0351b0a8" containerName="loki-ingester" Oct 30 11:28:23.585119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.584986 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5edc29a7-fe63-41b1-993c-712b0351b0a8" containerName="loki-ingester" Oct 30 11:28:23.585119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.585068 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="551b0f59-4902-49a5-a83f-85bad32dee9e" containerName="gateway" Oct 30 11:28:23.585119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.585082 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f6dbe1-349f-4ebb-b891-82e63db6d59a" containerName="loki-querier" Oct 30 11:28:23.585119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.585091 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="551b0f59-4902-49a5-a83f-85bad32dee9e" containerName="opa" Oct 30 11:28:23.585119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.585100 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="013b474e-89fc-4473-bf9e-e832bbd8b4a4" containerName="loki-index-gateway" Oct 30 11:28:23.585119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.585110 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="a517cbd2-329c-4c01-8f6f-955220592158" containerName="loki-distributor" Oct 30 11:28:23.585119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.585117 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="5edc29a7-fe63-41b1-993c-712b0351b0a8" containerName="loki-ingester" Oct 30 11:28:23.590182 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.590160 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" Oct 30 11:28:23.616961 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.616937 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-dockercfg-csr95\"" Oct 30 11:28:23.617146 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.616937 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-distributor-grpc\"" Oct 30 11:28:23.617230 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.617017 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 11:28:23.617230 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.617039 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-distributor-http\"" Oct 30 11:28:23.617230 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.617081 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 11:28:23.617358 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.617284 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ca-bundle\"" Oct 30 11:28:23.617358 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.617348 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-config\"" Oct 30 11:28:23.624397 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.624377 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf"] Oct 30 11:28:23.664402 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.664369 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 30 11:28:23.667615 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.667596 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:23.684034 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.683998 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:28:23.690350 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.690328 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 30 11:28:23.703106 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.703085 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/23be38ee-7767-4a92-a7cf-bf84b3690494-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-dtlmf\" (UID: \"23be38ee-7767-4a92-a7cf-bf84b3690494\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" Oct 30 11:28:23.703223 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.703118 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8x7p\" (UniqueName: \"kubernetes.io/projected/23be38ee-7767-4a92-a7cf-bf84b3690494-kube-api-access-g8x7p\") pod \"data-science-lokistack-distributor-58947679b7-dtlmf\" (UID: \"23be38ee-7767-4a92-a7cf-bf84b3690494\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" Oct 30 11:28:23.703272 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.703229 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23be38ee-7767-4a92-a7cf-bf84b3690494-config\") pod \"data-science-lokistack-distributor-58947679b7-dtlmf\" (UID: \"23be38ee-7767-4a92-a7cf-bf84b3690494\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" Oct 30 11:28:23.703272 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.703249 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23be38ee-7767-4a92-a7cf-bf84b3690494-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-dtlmf\" (UID: \"23be38ee-7767-4a92-a7cf-bf84b3690494\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" Oct 30 11:28:23.703370 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.703276 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/23be38ee-7767-4a92-a7cf-bf84b3690494-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-dtlmf\" (UID: \"23be38ee-7767-4a92-a7cf-bf84b3690494\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" Oct 30 11:28:23.705331 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.705315 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-lifecycle-secret\"" Oct 30 11:28:23.705643 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.705594 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-http\"" Oct 30 11:28:23.705760 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.705664 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-grpc\"" Oct 30 11:28:23.804246 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.804214 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/23be38ee-7767-4a92-a7cf-bf84b3690494-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-dtlmf\" (UID: \"23be38ee-7767-4a92-a7cf-bf84b3690494\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" Oct 30 11:28:23.804410 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.804256 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8x7p\" (UniqueName: \"kubernetes.io/projected/23be38ee-7767-4a92-a7cf-bf84b3690494-kube-api-access-g8x7p\") pod \"data-science-lokistack-distributor-58947679b7-dtlmf\" (UID: \"23be38ee-7767-4a92-a7cf-bf84b3690494\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" Oct 30 11:28:23.804410 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.804328 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23be38ee-7767-4a92-a7cf-bf84b3690494-config\") pod \"data-science-lokistack-distributor-58947679b7-dtlmf\" (UID: \"23be38ee-7767-4a92-a7cf-bf84b3690494\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" Oct 30 11:28:23.804410 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.804360 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23be38ee-7767-4a92-a7cf-bf84b3690494-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-dtlmf\" (UID: \"23be38ee-7767-4a92-a7cf-bf84b3690494\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" Oct 30 11:28:23.804410 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.804400 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/23be38ee-7767-4a92-a7cf-bf84b3690494-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-dtlmf\" (UID: \"23be38ee-7767-4a92-a7cf-bf84b3690494\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" Oct 30 11:28:23.805102 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.805071 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23be38ee-7767-4a92-a7cf-bf84b3690494-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-dtlmf\" (UID: \"23be38ee-7767-4a92-a7cf-bf84b3690494\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" Oct 30 11:28:23.805233 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.805212 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23be38ee-7767-4a92-a7cf-bf84b3690494-config\") pod \"data-science-lokistack-distributor-58947679b7-dtlmf\" (UID: \"23be38ee-7767-4a92-a7cf-bf84b3690494\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" Oct 30 11:28:23.806678 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.806649 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/23be38ee-7767-4a92-a7cf-bf84b3690494-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-dtlmf\" (UID: \"23be38ee-7767-4a92-a7cf-bf84b3690494\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" Oct 30 11:28:23.806763 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.806744 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/23be38ee-7767-4a92-a7cf-bf84b3690494-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-dtlmf\" (UID: \"23be38ee-7767-4a92-a7cf-bf84b3690494\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" Oct 30 11:28:23.829984 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.829942 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 30 11:28:23.830935 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.830905 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8x7p\" (UniqueName: \"kubernetes.io/projected/23be38ee-7767-4a92-a7cf-bf84b3690494-kube-api-access-g8x7p\") pod \"data-science-lokistack-distributor-58947679b7-dtlmf\" (UID: \"23be38ee-7767-4a92-a7cf-bf84b3690494\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" Oct 30 11:28:23.833709 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.833689 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:23.854461 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.854415 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:28:23.856665 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.856643 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 30 11:28:23.904742 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.904712 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" Oct 30 11:28:23.904887 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.904864 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:23.904959 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.904917 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:23.904959 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.904942 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:23.905062 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.904999 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:23.905138 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.905121 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-config\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:23.905188 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.905149 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvknn\" (UniqueName: \"kubernetes.io/projected/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-kube-api-access-wvknn\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:23.905443 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.905415 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-compactor-http\"" Oct 30 11:28:23.905558 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.905545 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-compactor-grpc\"" Oct 30 11:28:23.976856 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.976826 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 11:28:23.981130 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:23.981106 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:24.002191 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.002165 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 11:28:24.005698 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.005546 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:24.005698 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.005593 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/eebd0be1-9d9a-4524-8554-9e479a10b97a-test-loki-lifecycle-secret\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:24.005698 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.005632 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/eebd0be1-9d9a-4524-8554-9e479a10b97a-data-science-lokistack-compactor-grpc\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:24.005698 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.005695 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:24.006235 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.005732 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:24.006235 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.005758 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:24.006235 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.005792 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eebd0be1-9d9a-4524-8554-9e479a10b97a-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:24.006235 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.006138 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-config\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:24.006498 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.006481 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-http\"" Oct 30 11:28:24.006604 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.006578 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-grpc\"" Oct 30 11:28:24.006682 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.006661 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:24.007375 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.007325 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-config\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:24.007629 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.006170 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eebd0be1-9d9a-4524-8554-9e479a10b97a-config\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:24.007738 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.007670 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kbmz\" (UniqueName: \"kubernetes.io/projected/eebd0be1-9d9a-4524-8554-9e479a10b97a-kube-api-access-2kbmz\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:24.007805 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.007771 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvknn\" (UniqueName: \"kubernetes.io/projected/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-kube-api-access-wvknn\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:24.007872 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.007860 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/eebd0be1-9d9a-4524-8554-9e479a10b97a-data-science-lokistack-compactor-http\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:24.009217 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.009192 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:24.009609 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.009589 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:24.010987 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.010966 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:24.024701 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.024674 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:28:24.030639 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.030621 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvknn\" (UniqueName: \"kubernetes.io/projected/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-kube-api-access-wvknn\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:24.108902 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.108825 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eebd0be1-9d9a-4524-8554-9e479a10b97a-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:24.108902 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.108876 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eebd0be1-9d9a-4524-8554-9e479a10b97a-config\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:24.108902 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.108897 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kbmz\" (UniqueName: \"kubernetes.io/projected/eebd0be1-9d9a-4524-8554-9e479a10b97a-kube-api-access-2kbmz\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:24.109096 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.108945 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/eebd0be1-9d9a-4524-8554-9e479a10b97a-data-science-lokistack-compactor-http\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:24.109096 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.108983 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f972dd-9d39-497e-9486-6fbc0ef2b31f-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:24.109096 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.109009 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn89t\" (UniqueName: \"kubernetes.io/projected/36f972dd-9d39-497e-9486-6fbc0ef2b31f-kube-api-access-tn89t\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:24.109096 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.109040 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/36f972dd-9d39-497e-9486-6fbc0ef2b31f-test-loki-lifecycle-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:24.109096 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.109061 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/36f972dd-9d39-497e-9486-6fbc0ef2b31f-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:24.109363 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.109205 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/eebd0be1-9d9a-4524-8554-9e479a10b97a-test-loki-lifecycle-secret\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:24.109363 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.109294 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/eebd0be1-9d9a-4524-8554-9e479a10b97a-data-science-lokistack-compactor-grpc\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:24.109508 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.109394 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/36f972dd-9d39-497e-9486-6fbc0ef2b31f-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:24.109508 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.109465 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/36f972dd-9d39-497e-9486-6fbc0ef2b31f-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:24.109625 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.109613 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eebd0be1-9d9a-4524-8554-9e479a10b97a-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:24.109758 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.109733 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eebd0be1-9d9a-4524-8554-9e479a10b97a-config\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:24.111644 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.111418 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/eebd0be1-9d9a-4524-8554-9e479a10b97a-data-science-lokistack-compactor-http\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:24.111644 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.111610 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/eebd0be1-9d9a-4524-8554-9e479a10b97a-data-science-lokistack-compactor-grpc\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:24.111877 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.111857 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/eebd0be1-9d9a-4524-8554-9e479a10b97a-test-loki-lifecycle-secret\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:24.133778 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.133746 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kbmz\" (UniqueName: \"kubernetes.io/projected/eebd0be1-9d9a-4524-8554-9e479a10b97a-kube-api-access-2kbmz\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:24.155439 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.155399 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x"] Oct 30 11:28:24.161034 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.161014 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.178992 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.176926 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-ca-bundle\"" Oct 30 11:28:24.178992 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.177006 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\"" Oct 30 11:28:24.178992 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.178941 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x"] Oct 30 11:28:24.198934 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.198907 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:28:24.209859 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.209831 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/36f972dd-9d39-497e-9486-6fbc0ef2b31f-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:24.209963 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.209869 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e79ac819-335f-479d-9a1e-24590be62085-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-b7dfc66f6-4s62x\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.209963 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.209898 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/36f972dd-9d39-497e-9486-6fbc0ef2b31f-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:24.210072 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.210005 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/e79ac819-335f-479d-9a1e-24590be62085-lokistack-gateway\") pod \"data-science-lokistack-gateway-b7dfc66f6-4s62x\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.210147 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.210127 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/e79ac819-335f-479d-9a1e-24590be62085-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-b7dfc66f6-4s62x\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.210250 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.210231 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gszgn\" (UniqueName: \"kubernetes.io/projected/e79ac819-335f-479d-9a1e-24590be62085-kube-api-access-gszgn\") pod \"data-science-lokistack-gateway-b7dfc66f6-4s62x\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.210304 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.210272 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f972dd-9d39-497e-9486-6fbc0ef2b31f-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:24.210370 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.210311 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn89t\" (UniqueName: \"kubernetes.io/projected/36f972dd-9d39-497e-9486-6fbc0ef2b31f-kube-api-access-tn89t\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:24.210370 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.210344 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/e79ac819-335f-479d-9a1e-24590be62085-tls-secret\") pod \"data-science-lokistack-gateway-b7dfc66f6-4s62x\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.210499 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.210385 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/36f972dd-9d39-497e-9486-6fbc0ef2b31f-test-loki-lifecycle-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:24.210499 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.210413 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/36f972dd-9d39-497e-9486-6fbc0ef2b31f-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:24.210603 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.210498 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/e79ac819-335f-479d-9a1e-24590be62085-tenants\") pod \"data-science-lokistack-gateway-b7dfc66f6-4s62x\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.210603 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.210558 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/e79ac819-335f-479d-9a1e-24590be62085-rbac\") pod \"data-science-lokistack-gateway-b7dfc66f6-4s62x\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.210603 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.210588 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/e79ac819-335f-479d-9a1e-24590be62085-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-b7dfc66f6-4s62x\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.211118 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.210790 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/36f972dd-9d39-497e-9486-6fbc0ef2b31f-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:24.211118 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.210992 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f972dd-9d39-497e-9486-6fbc0ef2b31f-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:24.211377 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:28:24.211355 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23be38ee_7767_4a92_a7cf_bf84b3690494.slice/crio-f85567f22d13173d2c468233b08760ad12288154917fd701db59a919cdce68db WatchSource:0}: Error finding container f85567f22d13173d2c468233b08760ad12288154917fd701db59a919cdce68db: Status 404 returned error can't find the container with id f85567f22d13173d2c468233b08760ad12288154917fd701db59a919cdce68db Oct 30 11:28:24.211479 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.211420 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf"] Oct 30 11:28:24.212651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.212633 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/36f972dd-9d39-497e-9486-6fbc0ef2b31f-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:24.213248 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.213025 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/36f972dd-9d39-497e-9486-6fbc0ef2b31f-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:24.213381 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.213363 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/36f972dd-9d39-497e-9486-6fbc0ef2b31f-test-loki-lifecycle-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:24.232825 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.232807 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn89t\" (UniqueName: \"kubernetes.io/projected/36f972dd-9d39-497e-9486-6fbc0ef2b31f-kube-api-access-tn89t\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:24.244787 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.244769 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-client-http\"" Oct 30 11:28:24.244877 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.244791 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-http\"" Oct 30 11:28:24.244877 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.244802 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\"" Oct 30 11:28:24.244877 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.244833 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-dockercfg-rttgt\"" Oct 30 11:28:24.311567 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.311533 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/e79ac819-335f-479d-9a1e-24590be62085-lokistack-gateway\") pod \"data-science-lokistack-gateway-b7dfc66f6-4s62x\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.311567 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.311578 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/e79ac819-335f-479d-9a1e-24590be62085-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-b7dfc66f6-4s62x\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.311842 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.311698 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gszgn\" (UniqueName: \"kubernetes.io/projected/e79ac819-335f-479d-9a1e-24590be62085-kube-api-access-gszgn\") pod \"data-science-lokistack-gateway-b7dfc66f6-4s62x\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.311842 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.311738 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/e79ac819-335f-479d-9a1e-24590be62085-tls-secret\") pod \"data-science-lokistack-gateway-b7dfc66f6-4s62x\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.311842 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.311789 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/e79ac819-335f-479d-9a1e-24590be62085-tenants\") pod \"data-science-lokistack-gateway-b7dfc66f6-4s62x\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.312008 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.311856 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/e79ac819-335f-479d-9a1e-24590be62085-rbac\") pod \"data-science-lokistack-gateway-b7dfc66f6-4s62x\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.312008 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.311886 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/e79ac819-335f-479d-9a1e-24590be62085-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-b7dfc66f6-4s62x\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.312008 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.311934 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e79ac819-335f-479d-9a1e-24590be62085-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-b7dfc66f6-4s62x\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.312389 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.312359 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/e79ac819-335f-479d-9a1e-24590be62085-lokistack-gateway\") pod \"data-science-lokistack-gateway-b7dfc66f6-4s62x\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.312701 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.312674 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e79ac819-335f-479d-9a1e-24590be62085-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-b7dfc66f6-4s62x\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.312801 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.312680 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/e79ac819-335f-479d-9a1e-24590be62085-rbac\") pod \"data-science-lokistack-gateway-b7dfc66f6-4s62x\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.314314 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.314295 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/e79ac819-335f-479d-9a1e-24590be62085-tls-secret\") pod \"data-science-lokistack-gateway-b7dfc66f6-4s62x\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.314614 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.314592 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/e79ac819-335f-479d-9a1e-24590be62085-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-b7dfc66f6-4s62x\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.314770 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.314756 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/e79ac819-335f-479d-9a1e-24590be62085-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-b7dfc66f6-4s62x\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.314806 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.314751 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/e79ac819-335f-479d-9a1e-24590be62085-tenants\") pod \"data-science-lokistack-gateway-b7dfc66f6-4s62x\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.334891 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.334865 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gszgn\" (UniqueName: \"kubernetes.io/projected/e79ac819-335f-479d-9a1e-24590be62085-kube-api-access-gszgn\") pod \"data-science-lokistack-gateway-b7dfc66f6-4s62x\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.366499 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.366407 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:28:24.474944 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.474907 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:24.541854 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.541827 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:28:24.790105 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:28:24.790080 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79ac819_335f_479d_9a1e_24590be62085.slice/crio-077533c7bb80f4a3cbd52577f47f3fe13b67925616853e782937e0c598046024 WatchSource:0}: Error finding container 077533c7bb80f4a3cbd52577f47f3fe13b67925616853e782937e0c598046024: Status 404 returned error can't find the container with id 077533c7bb80f4a3cbd52577f47f3fe13b67925616853e782937e0c598046024 Oct 30 11:28:24.790463 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:24.790132 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x"] Oct 30 11:28:25.140883 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:25.140790 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" event={"ID":"e79ac819-335f-479d-9a1e-24590be62085","Type":"ContainerStarted","Data":"410bc3596004029993012562e90be905e9ced78990cac3a878219725ab43701a"} Oct 30 11:28:25.141054 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:25.140886 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" event={"ID":"e79ac819-335f-479d-9a1e-24590be62085","Type":"ContainerStarted","Data":"6440099cb827dd8cc7d0fb5041abb938397aba8f09dfca988666067624faf59f"} Oct 30 11:28:25.141054 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:25.140910 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" event={"ID":"e79ac819-335f-479d-9a1e-24590be62085","Type":"ContainerStarted","Data":"077533c7bb80f4a3cbd52577f47f3fe13b67925616853e782937e0c598046024"} Oct 30 11:28:25.141054 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:25.140933 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:25.141054 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:25.140954 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:25.142542 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:25.142506 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" event={"ID":"23be38ee-7767-4a92-a7cf-bf84b3690494","Type":"ContainerStarted","Data":"28653428c5505bf3de820dfd9d6fe81f2d4088b6cc0079f005a0a3e992925380"} Oct 30 11:28:25.142684 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:25.142560 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" Oct 30 11:28:25.142684 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:25.142576 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" event={"ID":"23be38ee-7767-4a92-a7cf-bf84b3690494","Type":"ContainerStarted","Data":"f85567f22d13173d2c468233b08760ad12288154917fd701db59a919cdce68db"} Oct 30 11:28:25.147286 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:25.147264 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:25.149966 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:25.149948 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:28:25.184497 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:25.184414 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" podStartSLOduration=1.184402735 podStartE2EDuration="1.184402735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:28:25.183739665 +0000 UTC m=+2595.685119151" watchObservedRunningTime="2026-10-30 11:28:25.184402735 +0000 UTC m=+2595.685782219" Oct 30 11:28:25.225790 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:25.225749 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" podStartSLOduration=2.225735287 podStartE2EDuration="2.225735287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:28:25.224494237 +0000 UTC m=+2595.725873750" watchObservedRunningTime="2026-10-30 11:28:25.225735287 +0000 UTC m=+2595.727114772" Oct 30 11:28:26.256636 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:28:26.256602 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:28:28.349878 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:28.349838 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:28.377840 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:28.377808 4289 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:28.450278 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:28.450245 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:28.450458 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:28.450392 4289 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:28.464448 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:28.464418 4289 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") DevicePath \"csi-2e669b6d2bbdfa3a6b46036bf22d4a7a6aba8888c2c130325f5aeb55a396d7c5\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:28.567513 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:28.567486 4289 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b48470aea0a0a5b9ac05ab05099d84e77041b5209a87fa2cabb2ba39e3c57c0c/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:28.592563 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:28.592534 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:28.795269 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:28.795233 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:29.092531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:29.092471 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 11:28:29.161553 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:29.161522 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"36f972dd-9d39-497e-9486-6fbc0ef2b31f","Type":"ContainerStarted","Data":"a0caa171ca4601f43627394728b6c1affed5b2d87a45bfc3ea3f4d35ed5981e8"} Oct 30 11:28:29.161704 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:29.161568 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"36f972dd-9d39-497e-9486-6fbc0ef2b31f","Type":"ContainerStarted","Data":"5800cedea907732a4a9f3028b8f6fe9860f4be26aef849815ae00dee657e1834"} Oct 30 11:28:29.161704 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:29.161640 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:29.206218 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:29.206158 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" podStartSLOduration=6.206141017 podStartE2EDuration="6.206141017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:28:29.206022463 +0000 UTC m=+2599.707402290" watchObservedRunningTime="2026-10-30 11:28:29.206141017 +0000 UTC m=+2599.707520506" Oct 30 11:28:35.213480 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:35.213444 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-7500c3e4-8eda-4feb-ad73-ef4f65175670\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c7411922d57e0ce3\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:35.241993 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:35.241967 4289 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-7500c3e4-8eda-4feb-ad73-ef4f65175670\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c7411922d57e0ce3\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:35.314047 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:35.314018 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-7500c3e4-8eda-4feb-ad73-ef4f65175670\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c7411922d57e0ce3\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:35.314215 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:35.314167 4289 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-7500c3e4-8eda-4feb-ad73-ef4f65175670\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c7411922d57e0ce3\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:35.328090 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:35.328065 4289 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-7500c3e4-8eda-4feb-ad73-ef4f65175670\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c7411922d57e0ce3\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") DevicePath \"csi-98bb03d3a5bccadf02ba39d0cce7ab5b127335095f1616eaf86b59584e7055e5\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:35.430360 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:35.430324 4289 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-7500c3e4-8eda-4feb-ad73-ef4f65175670\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c7411922d57e0ce3\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/1c19039a2608c8bacc8ebf206c5def5c3154a220ec38c0ab0294eb81856ec5cd/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:35.459855 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:35.459822 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-7500c3e4-8eda-4feb-ad73-ef4f65175670\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c7411922d57e0ce3\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:39.649578 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:39.649545 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-8b8fadbe-9e8f-4c46-82cd-d227d32b0511\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02dbd7be16ed1d85f\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:39.678304 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:39.678277 4289 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-8b8fadbe-9e8f-4c46-82cd-d227d32b0511\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02dbd7be16ed1d85f\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:39.750753 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:39.750726 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-8b8fadbe-9e8f-4c46-82cd-d227d32b0511\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02dbd7be16ed1d85f\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:39.750923 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:39.750823 4289 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-8b8fadbe-9e8f-4c46-82cd-d227d32b0511\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02dbd7be16ed1d85f\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:39.765313 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:39.765284 4289 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-8b8fadbe-9e8f-4c46-82cd-d227d32b0511\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02dbd7be16ed1d85f\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") DevicePath \"csi-1e2ea122b6c8a5785475645340f6c83dc8f3fc54c1c393443b28f4e4a3ad0d03\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:39.875312 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:39.875279 4289 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-8b8fadbe-9e8f-4c46-82cd-d227d32b0511\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02dbd7be16ed1d85f\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/f1ba4cd0a8ebf37ed4c6577964876a5661084ff0c33c7df67c3521fff1e49e3c/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:39.901576 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:39.901507 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-8b8fadbe-9e8f-4c46-82cd-d227d32b0511\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02dbd7be16ed1d85f\") pod \"data-science-lokistack-ingester-0\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:40.180675 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:40.180585 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:40.492561 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:40.492529 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 30 11:28:40.492899 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:28:40.492873 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1fcc3c_d5e7_494d_8c1f_da01485a222d.slice/crio-f0515eac796328578a8b2a3ab0dae41f7f096190c06d7e07efb69aac8fcb0d17 WatchSource:0}: Error finding container f0515eac796328578a8b2a3ab0dae41f7f096190c06d7e07efb69aac8fcb0d17: Status 404 returned error can't find the container with id f0515eac796328578a8b2a3ab0dae41f7f096190c06d7e07efb69aac8fcb0d17 Oct 30 11:28:41.213751 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:41.213713 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"8d1fcc3c-d5e7-494d-8c1f-da01485a222d","Type":"ContainerStarted","Data":"203e459e75ef5488b6bbd0292aa708c3b60638415f8c3e9ae51d48a701302f0e"} Oct 30 11:28:41.214135 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:41.213767 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"8d1fcc3c-d5e7-494d-8c1f-da01485a222d","Type":"ContainerStarted","Data":"f0515eac796328578a8b2a3ab0dae41f7f096190c06d7e07efb69aac8fcb0d17"} Oct 30 11:28:41.214135 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:41.213841 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:28:41.257175 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:41.257122 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podStartSLOduration=18.25708782 podStartE2EDuration="18.25708782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:28:41.256872898 +0000 UTC m=+2611.758252385" watchObservedRunningTime="2026-10-30 11:28:41.25708782 +0000 UTC m=+2611.758467310" Oct 30 11:28:45.500772 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:45.500738 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-7d33c086-8411-4c5e-bbb7-fd829bece632\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019deea708189b895\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:45.529646 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:45.529618 4289 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-7d33c086-8411-4c5e-bbb7-fd829bece632\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019deea708189b895\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:45.602100 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:45.602067 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-7d33c086-8411-4c5e-bbb7-fd829bece632\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019deea708189b895\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:45.602212 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:45.602176 4289 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-7d33c086-8411-4c5e-bbb7-fd829bece632\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019deea708189b895\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:45.616087 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:45.616063 4289 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-7d33c086-8411-4c5e-bbb7-fd829bece632\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019deea708189b895\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") DevicePath \"csi-147ed78c6f417cc29a86396ab08d558a13e0a27287da77e8d2e554e949970c87\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:45.719009 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:45.718979 4289 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-7d33c086-8411-4c5e-bbb7-fd829bece632\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019deea708189b895\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7df80f80ed0e4c658d21d8a82ac94fa2db4558ef72248dc2d4a0162a6c11c695/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:45.744400 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:45.744373 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-7d33c086-8411-4c5e-bbb7-fd829bece632\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019deea708189b895\") pod \"data-science-lokistack-compactor-0\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:46.048259 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:46.048228 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:46.151971 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:46.151946 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" Oct 30 11:28:46.343315 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:46.343287 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 30 11:28:46.343499 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:28:46.343476 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeebd0be1_9d9a_4524_8554_9e479a10b97a.slice/crio-36c6d3127b7118103f07c77187997db52261d064458d31b19f0b9fca42884f9e WatchSource:0}: Error finding container 36c6d3127b7118103f07c77187997db52261d064458d31b19f0b9fca42884f9e: Status 404 returned error can't find the container with id 36c6d3127b7118103f07c77187997db52261d064458d31b19f0b9fca42884f9e Oct 30 11:28:47.243697 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:47.243663 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"eebd0be1-9d9a-4524-8554-9e479a10b97a","Type":"ContainerStarted","Data":"45adefb8736453786fce91bbba45dcaddded6da10f6e4270270054d700a60f11"} Oct 30 11:28:47.244100 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:47.243720 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:28:47.244100 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:47.243739 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"eebd0be1-9d9a-4524-8554-9e479a10b97a","Type":"ContainerStarted","Data":"36c6d3127b7118103f07c77187997db52261d064458d31b19f0b9fca42884f9e"} Oct 30 11:28:47.299229 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:47.299178 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" podStartSLOduration=24.29916496 podStartE2EDuration="24.29916496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:28:47.286106267 +0000 UTC m=+2617.787485752" watchObservedRunningTime="2026-10-30 11:28:47.29916496 +0000 UTC m=+2617.800544446" Oct 30 11:28:50.172090 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:50.172059 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:28:56.257028 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:28:56.256993 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:28:58.511363 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:28:58.511325 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:29:02.225521 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:29:02.225482 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="8d1fcc3c-d5e7-494d-8c1f-da01485a222d" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 11:29:08.253703 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:29:08.253672 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:29:09.825870 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:29:09.825834 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:29:12.223957 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:29:12.223915 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="8d1fcc3c-d5e7-494d-8c1f-da01485a222d" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 11:29:22.223522 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:29:22.223479 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:29:26.258147 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:29:26.258107 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:29:56.256024 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:29:56.255986 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:29:58.630463 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:29:58.630410 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:30:26.257314 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:30:26.257276 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:30:56.260702 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:30:56.260611 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:30:58.767944 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:30:58.767900 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:31:19.640938 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:19.640901 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:31:19.808005 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:19.807970 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:31:26.256932 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:31:26.256894 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:31:32.484693 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:32.484656 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5964dd49df-kfbhf"] Oct 30 11:31:32.488825 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:32.488799 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5964dd49df-kfbhf" Oct 30 11:31:32.507504 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:32.507476 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-dockercfg-zzjcl\"" Oct 30 11:31:32.509715 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:32.509690 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5964dd49df-kfbhf"] Oct 30 11:31:32.520224 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:32.520201 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-6a6878d3\"" Oct 30 11:31:32.548698 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:32.548666 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65lwv\" (UniqueName: \"kubernetes.io/projected/5d6d9e7e-eec0-4132-97a0-a1a500df2de4-kube-api-access-65lwv\") pod \"data-science-usage-logs-collector-5964dd49df-kfbhf\" (UID: \"5d6d9e7e-eec0-4132-97a0-a1a500df2de4\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5964dd49df-kfbhf" Oct 30 11:31:32.548840 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:32.548715 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5d6d9e7e-eec0-4132-97a0-a1a500df2de4-otc-internal\") pod \"data-science-usage-logs-collector-5964dd49df-kfbhf\" (UID: \"5d6d9e7e-eec0-4132-97a0-a1a500df2de4\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5964dd49df-kfbhf" Oct 30 11:31:32.562547 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:32.562520 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:31:32.649937 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:32.649902 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5d6d9e7e-eec0-4132-97a0-a1a500df2de4-otc-internal\") pod \"data-science-usage-logs-collector-5964dd49df-kfbhf\" (UID: \"5d6d9e7e-eec0-4132-97a0-a1a500df2de4\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5964dd49df-kfbhf" Oct 30 11:31:32.650127 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:32.650103 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65lwv\" (UniqueName: \"kubernetes.io/projected/5d6d9e7e-eec0-4132-97a0-a1a500df2de4-kube-api-access-65lwv\") pod \"data-science-usage-logs-collector-5964dd49df-kfbhf\" (UID: \"5d6d9e7e-eec0-4132-97a0-a1a500df2de4\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5964dd49df-kfbhf" Oct 30 11:31:32.650681 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:32.650660 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5d6d9e7e-eec0-4132-97a0-a1a500df2de4-otc-internal\") pod \"data-science-usage-logs-collector-5964dd49df-kfbhf\" (UID: \"5d6d9e7e-eec0-4132-97a0-a1a500df2de4\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5964dd49df-kfbhf" Oct 30 11:31:32.673963 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:32.673935 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65lwv\" (UniqueName: \"kubernetes.io/projected/5d6d9e7e-eec0-4132-97a0-a1a500df2de4-kube-api-access-65lwv\") pod \"data-science-usage-logs-collector-5964dd49df-kfbhf\" (UID: \"5d6d9e7e-eec0-4132-97a0-a1a500df2de4\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5964dd49df-kfbhf" Oct 30 11:31:32.748829 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:32.748747 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:31:32.801345 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:32.801306 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5964dd49df-kfbhf" Oct 30 11:31:32.929211 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:32.929167 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:31:33.097452 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:33.097397 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5964dd49df-kfbhf"] Oct 30 11:31:33.099384 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:31:33.099358 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6d9e7e_eec0_4132_97a0_a1a500df2de4.slice/crio-e8f9b4be7547319783684de16a0451bf3860a35ca93c6744def4e844d8f10d83 WatchSource:0}: Error finding container e8f9b4be7547319783684de16a0451bf3860a35ca93c6744def4e844d8f10d83: Status 404 returned error can't find the container with id e8f9b4be7547319783684de16a0451bf3860a35ca93c6744def4e844d8f10d83 Oct 30 11:31:33.101751 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:33.101708 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:31:33.101840 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:33.101789 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:31:33.956331 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:33.956298 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5964dd49df-kfbhf" event={"ID":"5d6d9e7e-eec0-4132-97a0-a1a500df2de4","Type":"ContainerStarted","Data":"9185d5a74dd5383d4b5e6f4962c4d2e2c23643f300d1a378a5217f04f7d707c9"} Oct 30 11:31:33.956740 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:33.956347 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5964dd49df-kfbhf" event={"ID":"5d6d9e7e-eec0-4132-97a0-a1a500df2de4","Type":"ContainerStarted","Data":"e8f9b4be7547319783684de16a0451bf3860a35ca93c6744def4e844d8f10d83"} Oct 30 11:31:34.001320 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:34.001264 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5964dd49df-kfbhf" podStartSLOduration=2.001249355 podStartE2EDuration="2.001249355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:31:33.997377093 +0000 UTC m=+2784.498756589" watchObservedRunningTime="2026-10-30 11:31:34.001249355 +0000 UTC m=+2784.502628894" Oct 30 11:31:34.777085 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:34.777042 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:31:34.792747 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:34.792718 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 30 11:31:34.792914 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:34.792895 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="8d1fcc3c-d5e7-494d-8c1f-da01485a222d" containerName="loki-ingester" containerID="cri-o://203e459e75ef5488b6bbd0292aa708c3b60638415f8c3e9ae51d48a701302f0e" gracePeriod=600 Oct 30 11:31:34.805196 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:34.805163 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 30 11:31:34.811271 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:34.805402 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" podUID="eebd0be1-9d9a-4524-8554-9e479a10b97a" containerName="loki-compactor" containerID="cri-o://45adefb8736453786fce91bbba45dcaddded6da10f6e4270270054d700a60f11" gracePeriod=30 Oct 30 11:31:34.837809 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:34.837774 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 11:31:34.838037 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:34.838014 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" podUID="36f972dd-9d39-497e-9486-6fbc0ef2b31f" containerName="loki-index-gateway" containerID="cri-o://a0caa171ca4601f43627394728b6c1affed5b2d87a45bfc3ea3f4d35ed5981e8" gracePeriod=30 Oct 30 11:31:34.868358 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:34.868332 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x"] Oct 30 11:31:34.868574 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:34.868550 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" podUID="e79ac819-335f-479d-9a1e-24590be62085" containerName="gateway" containerID="cri-o://6440099cb827dd8cc7d0fb5041abb938397aba8f09dfca988666067624faf59f" gracePeriod=30 Oct 30 11:31:34.868679 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:34.868560 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" podUID="e79ac819-335f-479d-9a1e-24590be62085" containerName="opa" containerID="cri-o://410bc3596004029993012562e90be905e9ced78990cac3a878219725ab43701a" gracePeriod=30 Oct 30 11:31:34.873235 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:34.872885 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf"] Oct 30 11:31:34.873235 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:34.873117 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" podUID="23be38ee-7767-4a92-a7cf-bf84b3690494" containerName="loki-distributor" containerID="cri-o://28653428c5505bf3de820dfd9d6fe81f2d4088b6cc0079f005a0a3e992925380" gracePeriod=30 Oct 30 11:31:34.965167 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:34.965068 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:31:34.970710 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:34.970680 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5964dd49df-kfbhf"] Oct 30 11:31:34.972860 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:34.972829 4289 generic.go:356] "Generic (PLEG): container finished" podID="36f972dd-9d39-497e-9486-6fbc0ef2b31f" containerID="a0caa171ca4601f43627394728b6c1affed5b2d87a45bfc3ea3f4d35ed5981e8" exitCode=0 Oct 30 11:31:34.973062 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:34.973034 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"36f972dd-9d39-497e-9486-6fbc0ef2b31f","Type":"ContainerDied","Data":"a0caa171ca4601f43627394728b6c1affed5b2d87a45bfc3ea3f4d35ed5981e8"} Oct 30 11:31:34.973237 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:34.973198 4289 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/data-science-usage-logs-collector-5964dd49df-kfbhf" secret="" err="secret \"data-science-usage-logs-collector-dockercfg-zzjcl\" not found" Oct 30 11:31:35.071480 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:31:35.071373 4289 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-usage-logs-collector-6a6878d3: configmap "data-science-usage-logs-collector-6a6878d3" not found Oct 30 11:31:35.071480 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:31:35.071462 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5d6d9e7e-eec0-4132-97a0-a1a500df2de4-otc-internal podName:5d6d9e7e-eec0-4132-97a0-a1a500df2de4 nodeName:}" failed. No retries permitted until 2026-10-30 11:31:35.571438955 +0000 UTC m=+2786.072818483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/5d6d9e7e-eec0-4132-97a0-a1a500df2de4-otc-internal") pod "data-science-usage-logs-collector-5964dd49df-kfbhf" (UID: "5d6d9e7e-eec0-4132-97a0-a1a500df2de4") : configmap "data-science-usage-logs-collector-6a6878d3" not found Oct 30 11:31:35.131332 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.131290 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:31:35.141260 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.141216 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" podUID="e79ac819-335f-479d-9a1e-24590be62085" containerName="opa" probeResult="failure" output="Get \"https://10.134.0.101:8083/ready\": dial tcp 10.134.0.101:8083: connect: connection refused" Oct 30 11:31:35.145402 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.145374 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" podUID="e79ac819-335f-479d-9a1e-24590be62085" containerName="gateway" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 11:31:35.169370 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.169352 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:31:35.272947 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.272918 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36f972dd-9d39-497e-9486-6fbc0ef2b31f-data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/36f972dd-9d39-497e-9486-6fbc0ef2b31f-data-science-lokistack-index-gateway-http\") pod \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " Oct 30 11:31:35.273093 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.272983 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f972dd-9d39-497e-9486-6fbc0ef2b31f-kube-api-access-tn89t\" (UniqueName: \"kubernetes.io/projected/36f972dd-9d39-497e-9486-6fbc0ef2b31f-kube-api-access-tn89t\") pod \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " Oct 30 11:31:35.273093 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.273078 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") pod \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " Oct 30 11:31:35.273208 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.273110 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36f972dd-9d39-497e-9486-6fbc0ef2b31f-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f972dd-9d39-497e-9486-6fbc0ef2b31f-data-science-lokistack-ca-bundle\") pod \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " Oct 30 11:31:35.273208 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.273141 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36f972dd-9d39-497e-9486-6fbc0ef2b31f-data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/36f972dd-9d39-497e-9486-6fbc0ef2b31f-data-science-lokistack-index-gateway-grpc\") pod \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " Oct 30 11:31:35.273208 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.273168 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36f972dd-9d39-497e-9486-6fbc0ef2b31f-config\" (UniqueName: \"kubernetes.io/configmap/36f972dd-9d39-497e-9486-6fbc0ef2b31f-config\") pod \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " Oct 30 11:31:35.273366 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.273344 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36f972dd-9d39-497e-9486-6fbc0ef2b31f-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/36f972dd-9d39-497e-9486-6fbc0ef2b31f-test-loki-lifecycle-secret\") pod \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\" (UID: \"36f972dd-9d39-497e-9486-6fbc0ef2b31f\") " Oct 30 11:31:35.273529 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.273506 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f972dd-9d39-497e-9486-6fbc0ef2b31f-data-science-lokistack-ca-bundle" pod "36f972dd-9d39-497e-9486-6fbc0ef2b31f" (UID: "36f972dd-9d39-497e-9486-6fbc0ef2b31f"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:31:35.273731 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.273711 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f972dd-9d39-497e-9486-6fbc0ef2b31f-data-science-lokistack-ca-bundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:35.273971 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.273932 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f972dd-9d39-497e-9486-6fbc0ef2b31f-config" pod "36f972dd-9d39-497e-9486-6fbc0ef2b31f" (UID: "36f972dd-9d39-497e-9486-6fbc0ef2b31f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:31:35.276272 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.276252 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3" pod "36f972dd-9d39-497e-9486-6fbc0ef2b31f" (UID: "36f972dd-9d39-497e-9486-6fbc0ef2b31f"). InnerVolumeSpecName "pvc-c30998fa-a6d9-4900-9b47-20d22faf37db". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 11:31:35.276916 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.276888 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36f972dd-9d39-497e-9486-6fbc0ef2b31f-data-science-lokistack-index-gateway-grpc" pod "36f972dd-9d39-497e-9486-6fbc0ef2b31f" (UID: "36f972dd-9d39-497e-9486-6fbc0ef2b31f"). InnerVolumeSpecName "data-science-lokistack-index-gateway-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:35.281757 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.281736 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f972dd-9d39-497e-9486-6fbc0ef2b31f-kube-api-access-tn89t" pod "36f972dd-9d39-497e-9486-6fbc0ef2b31f" (UID: "36f972dd-9d39-497e-9486-6fbc0ef2b31f"). InnerVolumeSpecName "kube-api-access-tn89t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:31:35.303811 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.303738 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36f972dd-9d39-497e-9486-6fbc0ef2b31f-data-science-lokistack-index-gateway-http" pod "36f972dd-9d39-497e-9486-6fbc0ef2b31f" (UID: "36f972dd-9d39-497e-9486-6fbc0ef2b31f"). InnerVolumeSpecName "data-science-lokistack-index-gateway-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:35.303811 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.303772 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36f972dd-9d39-497e-9486-6fbc0ef2b31f-test-loki-lifecycle-secret" pod "36f972dd-9d39-497e-9486-6fbc0ef2b31f" (UID: "36f972dd-9d39-497e-9486-6fbc0ef2b31f"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:35.375314 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.375280 4289 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/36f972dd-9d39-497e-9486-6fbc0ef2b31f-test-loki-lifecycle-secret\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:35.375475 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.375322 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/36f972dd-9d39-497e-9486-6fbc0ef2b31f-data-science-lokistack-index-gateway-http\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:35.375475 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.375338 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn89t\" (UniqueName: \"kubernetes.io/projected/36f972dd-9d39-497e-9486-6fbc0ef2b31f-kube-api-access-tn89t\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:35.375475 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.375375 4289 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") on node \"ip-10-0-137-242.ec2.internal\" " Oct 30 11:31:35.375475 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.375391 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/36f972dd-9d39-497e-9486-6fbc0ef2b31f-data-science-lokistack-index-gateway-grpc\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:35.375475 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.375405 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/36f972dd-9d39-497e-9486-6fbc0ef2b31f-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:35.388449 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.388408 4289 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-c30998fa-a6d9-4900-9b47-20d22faf37db" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3") on node "ip-10-0-137-242.ec2.internal" Oct 30 11:31:35.449098 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.449074 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:31:35.476407 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.476385 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-02dbd7be16ed1d85f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02dbd7be16ed1d85f\") pod \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " Oct 30 11:31:35.476559 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.476441 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-data-science-lokistack-ingester-grpc\") pod \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " Oct 30 11:31:35.476559 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.476495 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-config\" (UniqueName: \"kubernetes.io/configmap/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-config\") pod \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " Oct 30 11:31:35.476559 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.476532 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-test-loki-lifecycle-secret\") pod \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " Oct 30 11:31:35.478096 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.476563 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-data-science-lokistack-ingester-http\") pod \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " Oct 30 11:31:35.478096 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.476622 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-data-science-lokistack-ca-bundle\") pod \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " Oct 30 11:31:35.478096 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.476698 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c7411922d57e0ce3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c7411922d57e0ce3\") pod \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " Oct 30 11:31:35.478096 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.476735 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-kube-api-access-wvknn\" (UniqueName: \"kubernetes.io/projected/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-kube-api-access-wvknn\") pod \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\" (UID: \"8d1fcc3c-d5e7-494d-8c1f-da01485a222d\") " Oct 30 11:31:35.478096 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.476994 4289 reconciler_common.go:299] "Volume detached for volume \"pvc-c30998fa-a6d9-4900-9b47-20d22faf37db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087ae7af29064f8d3\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"csi-2e669b6d2bbdfa3a6b46036bf22d4a7a6aba8888c2c130325f5aeb55a396d7c5\"" Oct 30 11:31:35.478096 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.477002 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-config" pod "8d1fcc3c-d5e7-494d-8c1f-da01485a222d" (UID: "8d1fcc3c-d5e7-494d-8c1f-da01485a222d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:31:35.478096 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.477018 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-data-science-lokistack-ca-bundle" pod "8d1fcc3c-d5e7-494d-8c1f-da01485a222d" (UID: "8d1fcc3c-d5e7-494d-8c1f-da01485a222d"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:31:35.479194 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.479167 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-data-science-lokistack-ingester-grpc" pod "8d1fcc3c-d5e7-494d-8c1f-da01485a222d" (UID: "8d1fcc3c-d5e7-494d-8c1f-da01485a222d"). InnerVolumeSpecName "data-science-lokistack-ingester-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:35.479660 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.479264 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-data-science-lokistack-ingester-http" pod "8d1fcc3c-d5e7-494d-8c1f-da01485a222d" (UID: "8d1fcc3c-d5e7-494d-8c1f-da01485a222d"). InnerVolumeSpecName "data-science-lokistack-ingester-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:35.479660 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.479501 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-test-loki-lifecycle-secret" pod "8d1fcc3c-d5e7-494d-8c1f-da01485a222d" (UID: "8d1fcc3c-d5e7-494d-8c1f-da01485a222d"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:35.479660 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.479635 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-kube-api-access-wvknn" pod "8d1fcc3c-d5e7-494d-8c1f-da01485a222d" (UID: "8d1fcc3c-d5e7-494d-8c1f-da01485a222d"). InnerVolumeSpecName "kube-api-access-wvknn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:31:35.479962 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.479930 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-02dbd7be16ed1d85f" pod "8d1fcc3c-d5e7-494d-8c1f-da01485a222d" (UID: "8d1fcc3c-d5e7-494d-8c1f-da01485a222d"). InnerVolumeSpecName "pvc-8b8fadbe-9e8f-4c46-82cd-d227d32b0511". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 11:31:35.479962 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.479945 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0c7411922d57e0ce3" pod "8d1fcc3c-d5e7-494d-8c1f-da01485a222d" (UID: "8d1fcc3c-d5e7-494d-8c1f-da01485a222d"). InnerVolumeSpecName "pvc-7500c3e4-8eda-4feb-ad73-ef4f65175670". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 11:31:35.577520 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.577419 4289 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-8b8fadbe-9e8f-4c46-82cd-d227d32b0511\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02dbd7be16ed1d85f\") on node \"ip-10-0-137-242.ec2.internal\" " Oct 30 11:31:35.577520 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.577472 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-data-science-lokistack-ingester-grpc\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:35.577520 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:31:35.577480 4289 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-usage-logs-collector-6a6878d3: configmap "data-science-usage-logs-collector-6a6878d3" not found Oct 30 11:31:35.577713 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:31:35.577555 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5d6d9e7e-eec0-4132-97a0-a1a500df2de4-otc-internal podName:5d6d9e7e-eec0-4132-97a0-a1a500df2de4 nodeName:}" failed. No retries permitted until 2026-10-30 11:31:36.577539127 +0000 UTC m=+2787.078918594 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/5d6d9e7e-eec0-4132-97a0-a1a500df2de4-otc-internal") pod "data-science-usage-logs-collector-5964dd49df-kfbhf" (UID: "5d6d9e7e-eec0-4132-97a0-a1a500df2de4") : configmap "data-science-usage-logs-collector-6a6878d3" not found Oct 30 11:31:35.577713 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.577488 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:35.577713 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.577593 4289 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-test-loki-lifecycle-secret\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:35.577713 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.577612 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-data-science-lokistack-ingester-http\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:35.577713 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.577623 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-data-science-lokistack-ca-bundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:35.577713 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.577649 4289 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-7500c3e4-8eda-4feb-ad73-ef4f65175670\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c7411922d57e0ce3\") on node \"ip-10-0-137-242.ec2.internal\" " Oct 30 11:31:35.577713 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.577660 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvknn\" (UniqueName: \"kubernetes.io/projected/8d1fcc3c-d5e7-494d-8c1f-da01485a222d-kube-api-access-wvknn\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:35.586619 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.586601 4289 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-8b8fadbe-9e8f-4c46-82cd-d227d32b0511" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-02dbd7be16ed1d85f") on node "ip-10-0-137-242.ec2.internal" Oct 30 11:31:35.599820 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.599805 4289 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-7500c3e4-8eda-4feb-ad73-ef4f65175670" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0c7411922d57e0ce3") on node "ip-10-0-137-242.ec2.internal" Oct 30 11:31:35.678502 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.678477 4289 reconciler_common.go:299] "Volume detached for volume \"pvc-8b8fadbe-9e8f-4c46-82cd-d227d32b0511\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02dbd7be16ed1d85f\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"csi-1e2ea122b6c8a5785475645340f6c83dc8f3fc54c1c393443b28f4e4a3ad0d03\"" Oct 30 11:31:35.678659 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.678505 4289 reconciler_common.go:299] "Volume detached for volume \"pvc-7500c3e4-8eda-4feb-ad73-ef4f65175670\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c7411922d57e0ce3\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"csi-98bb03d3a5bccadf02ba39d0cce7ab5b127335095f1616eaf86b59584e7055e5\"" Oct 30 11:31:35.761359 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.761333 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:31:35.779519 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.779488 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eebd0be1-9d9a-4524-8554-9e479a10b97a-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eebd0be1-9d9a-4524-8554-9e479a10b97a-data-science-lokistack-ca-bundle\") pod \"eebd0be1-9d9a-4524-8554-9e479a10b97a\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " Oct 30 11:31:35.779693 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.779670 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eebd0be1-9d9a-4524-8554-9e479a10b97a-data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/eebd0be1-9d9a-4524-8554-9e479a10b97a-data-science-lokistack-compactor-grpc\") pod \"eebd0be1-9d9a-4524-8554-9e479a10b97a\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " Oct 30 11:31:35.779754 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.779727 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eebd0be1-9d9a-4524-8554-9e479a10b97a-config\" (UniqueName: \"kubernetes.io/configmap/eebd0be1-9d9a-4524-8554-9e479a10b97a-config\") pod \"eebd0be1-9d9a-4524-8554-9e479a10b97a\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " Oct 30 11:31:35.779814 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.779758 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eebd0be1-9d9a-4524-8554-9e479a10b97a-data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/eebd0be1-9d9a-4524-8554-9e479a10b97a-data-science-lokistack-compactor-http\") pod \"eebd0be1-9d9a-4524-8554-9e479a10b97a\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " Oct 30 11:31:35.779879 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.779809 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eebd0be1-9d9a-4524-8554-9e479a10b97a-data-science-lokistack-ca-bundle" pod "eebd0be1-9d9a-4524-8554-9e479a10b97a" (UID: "eebd0be1-9d9a-4524-8554-9e479a10b97a"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:31:35.779937 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.779915 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-019deea708189b895\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019deea708189b895\") pod \"eebd0be1-9d9a-4524-8554-9e479a10b97a\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " Oct 30 11:31:35.779995 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.779971 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eebd0be1-9d9a-4524-8554-9e479a10b97a-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/eebd0be1-9d9a-4524-8554-9e479a10b97a-test-loki-lifecycle-secret\") pod \"eebd0be1-9d9a-4524-8554-9e479a10b97a\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " Oct 30 11:31:35.780054 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.780001 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eebd0be1-9d9a-4524-8554-9e479a10b97a-kube-api-access-2kbmz\" (UniqueName: \"kubernetes.io/projected/eebd0be1-9d9a-4524-8554-9e479a10b97a-kube-api-access-2kbmz\") pod \"eebd0be1-9d9a-4524-8554-9e479a10b97a\" (UID: \"eebd0be1-9d9a-4524-8554-9e479a10b97a\") " Oct 30 11:31:35.780356 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.780161 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eebd0be1-9d9a-4524-8554-9e479a10b97a-config" pod "eebd0be1-9d9a-4524-8554-9e479a10b97a" (UID: "eebd0be1-9d9a-4524-8554-9e479a10b97a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:31:35.781132 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.780709 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eebd0be1-9d9a-4524-8554-9e479a10b97a-data-science-lokistack-ca-bundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:35.781132 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.780765 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eebd0be1-9d9a-4524-8554-9e479a10b97a-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:35.782703 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.782676 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eebd0be1-9d9a-4524-8554-9e479a10b97a-data-science-lokistack-compactor-grpc" pod "eebd0be1-9d9a-4524-8554-9e479a10b97a" (UID: "eebd0be1-9d9a-4524-8554-9e479a10b97a"). InnerVolumeSpecName "data-science-lokistack-compactor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:35.782913 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.782879 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eebd0be1-9d9a-4524-8554-9e479a10b97a-data-science-lokistack-compactor-http" pod "eebd0be1-9d9a-4524-8554-9e479a10b97a" (UID: "eebd0be1-9d9a-4524-8554-9e479a10b97a"). InnerVolumeSpecName "data-science-lokistack-compactor-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:35.783076 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.783053 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eebd0be1-9d9a-4524-8554-9e479a10b97a-test-loki-lifecycle-secret" pod "eebd0be1-9d9a-4524-8554-9e479a10b97a" (UID: "eebd0be1-9d9a-4524-8554-9e479a10b97a"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:35.783076 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.783068 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eebd0be1-9d9a-4524-8554-9e479a10b97a-kube-api-access-2kbmz" pod "eebd0be1-9d9a-4524-8554-9e479a10b97a" (UID: "eebd0be1-9d9a-4524-8554-9e479a10b97a"). InnerVolumeSpecName "kube-api-access-2kbmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:31:35.783257 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.783235 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-019deea708189b895" pod "eebd0be1-9d9a-4524-8554-9e479a10b97a" (UID: "eebd0be1-9d9a-4524-8554-9e479a10b97a"). InnerVolumeSpecName "pvc-7d33c086-8411-4c5e-bbb7-fd829bece632". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 11:31:35.881497 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.881470 4289 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-7d33c086-8411-4c5e-bbb7-fd829bece632\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019deea708189b895\") on node \"ip-10-0-137-242.ec2.internal\" " Oct 30 11:31:35.881497 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.881499 4289 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/eebd0be1-9d9a-4524-8554-9e479a10b97a-test-loki-lifecycle-secret\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:35.881678 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.881510 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kbmz\" (UniqueName: \"kubernetes.io/projected/eebd0be1-9d9a-4524-8554-9e479a10b97a-kube-api-access-2kbmz\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:35.881678 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.881522 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/eebd0be1-9d9a-4524-8554-9e479a10b97a-data-science-lokistack-compactor-grpc\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:35.881678 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.881538 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/eebd0be1-9d9a-4524-8554-9e479a10b97a-data-science-lokistack-compactor-http\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:35.890459 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.890425 4289 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-7d33c086-8411-4c5e-bbb7-fd829bece632" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-019deea708189b895") on node "ip-10-0-137-242.ec2.internal" Oct 30 11:31:35.979618 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.979584 4289 generic.go:356] "Generic (PLEG): container finished" podID="e79ac819-335f-479d-9a1e-24590be62085" containerID="410bc3596004029993012562e90be905e9ced78990cac3a878219725ab43701a" exitCode=0 Oct 30 11:31:35.980012 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.979663 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" event={"ID":"e79ac819-335f-479d-9a1e-24590be62085","Type":"ContainerDied","Data":"410bc3596004029993012562e90be905e9ced78990cac3a878219725ab43701a"} Oct 30 11:31:35.980984 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.980953 4289 generic.go:356] "Generic (PLEG): container finished" podID="8d1fcc3c-d5e7-494d-8c1f-da01485a222d" containerID="203e459e75ef5488b6bbd0292aa708c3b60638415f8c3e9ae51d48a701302f0e" exitCode=0 Oct 30 11:31:35.981145 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.981009 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 11:31:35.981145 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.981047 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"8d1fcc3c-d5e7-494d-8c1f-da01485a222d","Type":"ContainerDied","Data":"203e459e75ef5488b6bbd0292aa708c3b60638415f8c3e9ae51d48a701302f0e"} Oct 30 11:31:35.981145 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.981093 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"8d1fcc3c-d5e7-494d-8c1f-da01485a222d","Type":"ContainerDied","Data":"f0515eac796328578a8b2a3ab0dae41f7f096190c06d7e07efb69aac8fcb0d17"} Oct 30 11:31:35.981145 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.981115 4289 scope.go:122] "RemoveContainer" containerID="203e459e75ef5488b6bbd0292aa708c3b60638415f8c3e9ae51d48a701302f0e" Oct 30 11:31:35.982045 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.982021 4289 reconciler_common.go:299] "Volume detached for volume \"pvc-7d33c086-8411-4c5e-bbb7-fd829bece632\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-019deea708189b895\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"csi-147ed78c6f417cc29a86396ab08d558a13e0a27287da77e8d2e554e949970c87\"" Oct 30 11:31:35.982717 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.982616 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"36f972dd-9d39-497e-9486-6fbc0ef2b31f","Type":"ContainerDied","Data":"5800cedea907732a4a9f3028b8f6fe9860f4be26aef849815ae00dee657e1834"} Oct 30 11:31:35.982717 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.982619 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 11:31:35.984272 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.984179 4289 generic.go:356] "Generic (PLEG): container finished" podID="eebd0be1-9d9a-4524-8554-9e479a10b97a" containerID="45adefb8736453786fce91bbba45dcaddded6da10f6e4270270054d700a60f11" exitCode=0 Oct 30 11:31:35.984272 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.984213 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 11:31:35.984449 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.984210 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"eebd0be1-9d9a-4524-8554-9e479a10b97a","Type":"ContainerDied","Data":"45adefb8736453786fce91bbba45dcaddded6da10f6e4270270054d700a60f11"} Oct 30 11:31:35.984449 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.984369 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"eebd0be1-9d9a-4524-8554-9e479a10b97a","Type":"ContainerDied","Data":"36c6d3127b7118103f07c77187997db52261d064458d31b19f0b9fca42884f9e"} Oct 30 11:31:35.984632 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.984507 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5964dd49df-kfbhf" podUID="5d6d9e7e-eec0-4132-97a0-a1a500df2de4" containerName="otc-container" containerID="cri-o://9185d5a74dd5383d4b5e6f4962c4d2e2c23643f300d1a378a5217f04f7d707c9" gracePeriod=30 Oct 30 11:31:35.991905 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.991885 4289 scope.go:122] "RemoveContainer" containerID="203e459e75ef5488b6bbd0292aa708c3b60638415f8c3e9ae51d48a701302f0e" Oct 30 11:31:35.992175 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:31:35.992157 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"203e459e75ef5488b6bbd0292aa708c3b60638415f8c3e9ae51d48a701302f0e\": container with ID starting with 203e459e75ef5488b6bbd0292aa708c3b60638415f8c3e9ae51d48a701302f0e not found: ID does not exist" containerID="203e459e75ef5488b6bbd0292aa708c3b60638415f8c3e9ae51d48a701302f0e" Oct 30 11:31:35.992244 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.992181 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"203e459e75ef5488b6bbd0292aa708c3b60638415f8c3e9ae51d48a701302f0e"} err="failed to get container status \"203e459e75ef5488b6bbd0292aa708c3b60638415f8c3e9ae51d48a701302f0e\": rpc error: code = NotFound desc = could not find container \"203e459e75ef5488b6bbd0292aa708c3b60638415f8c3e9ae51d48a701302f0e\": container with ID starting with 203e459e75ef5488b6bbd0292aa708c3b60638415f8c3e9ae51d48a701302f0e not found: ID does not exist" Oct 30 11:31:35.992244 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:35.992197 4289 scope.go:122] "RemoveContainer" containerID="a0caa171ca4601f43627394728b6c1affed5b2d87a45bfc3ea3f4d35ed5981e8" Oct 30 11:31:36.033096 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.033064 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 30 11:31:36.036934 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.036908 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 30 11:31:36.073689 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.073658 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 11:31:36.076879 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.076831 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 11:31:36.095962 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.095937 4289 scope.go:122] "RemoveContainer" containerID="45adefb8736453786fce91bbba45dcaddded6da10f6e4270270054d700a60f11" Oct 30 11:31:36.105420 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.105400 4289 scope.go:122] "RemoveContainer" containerID="45adefb8736453786fce91bbba45dcaddded6da10f6e4270270054d700a60f11" Oct 30 11:31:36.105752 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:31:36.105722 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45adefb8736453786fce91bbba45dcaddded6da10f6e4270270054d700a60f11\": container with ID starting with 45adefb8736453786fce91bbba45dcaddded6da10f6e4270270054d700a60f11 not found: ID does not exist" containerID="45adefb8736453786fce91bbba45dcaddded6da10f6e4270270054d700a60f11" Oct 30 11:31:36.105837 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.105757 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45adefb8736453786fce91bbba45dcaddded6da10f6e4270270054d700a60f11"} err="failed to get container status \"45adefb8736453786fce91bbba45dcaddded6da10f6e4270270054d700a60f11\": rpc error: code = NotFound desc = could not find container \"45adefb8736453786fce91bbba45dcaddded6da10f6e4270270054d700a60f11\": container with ID starting with 45adefb8736453786fce91bbba45dcaddded6da10f6e4270270054d700a60f11 not found: ID does not exist" Oct 30 11:31:36.131416 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.131390 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:31:36.150955 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.150922 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" podUID="23be38ee-7767-4a92-a7cf-bf84b3690494" containerName="loki-distributor" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 11:31:36.183722 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.183693 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e79ac819-335f-479d-9a1e-24590be62085-tenants\" (UniqueName: \"kubernetes.io/secret/e79ac819-335f-479d-9a1e-24590be62085-tenants\") pod \"e79ac819-335f-479d-9a1e-24590be62085\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " Oct 30 11:31:36.183865 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.183739 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e79ac819-335f-479d-9a1e-24590be62085-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/e79ac819-335f-479d-9a1e-24590be62085-lokistack-gateway\") pod \"e79ac819-335f-479d-9a1e-24590be62085\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " Oct 30 11:31:36.183865 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.183761 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e79ac819-335f-479d-9a1e-24590be62085-kube-api-access-gszgn\" (UniqueName: \"kubernetes.io/projected/e79ac819-335f-479d-9a1e-24590be62085-kube-api-access-gszgn\") pod \"e79ac819-335f-479d-9a1e-24590be62085\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " Oct 30 11:31:36.183865 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.183799 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e79ac819-335f-479d-9a1e-24590be62085-tls-secret\" (UniqueName: \"kubernetes.io/projected/e79ac819-335f-479d-9a1e-24590be62085-tls-secret\") pod \"e79ac819-335f-479d-9a1e-24590be62085\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " Oct 30 11:31:36.183865 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.183838 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e79ac819-335f-479d-9a1e-24590be62085-data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/e79ac819-335f-479d-9a1e-24590be62085-data-science-lokistack-gateway-ca-bundle\") pod \"e79ac819-335f-479d-9a1e-24590be62085\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " Oct 30 11:31:36.184109 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.183885 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e79ac819-335f-479d-9a1e-24590be62085-rbac\" (UniqueName: \"kubernetes.io/configmap/e79ac819-335f-479d-9a1e-24590be62085-rbac\") pod \"e79ac819-335f-479d-9a1e-24590be62085\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " Oct 30 11:31:36.184109 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.183929 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e79ac819-335f-479d-9a1e-24590be62085-data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/e79ac819-335f-479d-9a1e-24590be62085-data-science-lokistack-gateway-client-http\") pod \"e79ac819-335f-479d-9a1e-24590be62085\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " Oct 30 11:31:36.184109 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.183966 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e79ac819-335f-479d-9a1e-24590be62085-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e79ac819-335f-479d-9a1e-24590be62085-data-science-lokistack-ca-bundle\") pod \"e79ac819-335f-479d-9a1e-24590be62085\" (UID: \"e79ac819-335f-479d-9a1e-24590be62085\") " Oct 30 11:31:36.184622 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.184593 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e79ac819-335f-479d-9a1e-24590be62085-data-science-lokistack-ca-bundle" pod "e79ac819-335f-479d-9a1e-24590be62085" (UID: "e79ac819-335f-479d-9a1e-24590be62085"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:31:36.187162 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.187134 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79ac819-335f-479d-9a1e-24590be62085-kube-api-access-gszgn" pod "e79ac819-335f-479d-9a1e-24590be62085" (UID: "e79ac819-335f-479d-9a1e-24590be62085"). InnerVolumeSpecName "kube-api-access-gszgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:31:36.187266 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.187159 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79ac819-335f-479d-9a1e-24590be62085-tls-secret" pod "e79ac819-335f-479d-9a1e-24590be62085" (UID: "e79ac819-335f-479d-9a1e-24590be62085"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:31:36.187266 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.187233 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e79ac819-335f-479d-9a1e-24590be62085-data-science-lokistack-gateway-client-http" pod "e79ac819-335f-479d-9a1e-24590be62085" (UID: "e79ac819-335f-479d-9a1e-24590be62085"). InnerVolumeSpecName "data-science-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:36.190287 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.190255 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e79ac819-335f-479d-9a1e-24590be62085-rbac" pod "e79ac819-335f-479d-9a1e-24590be62085" (UID: "e79ac819-335f-479d-9a1e-24590be62085"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:31:36.190445 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.190376 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e79ac819-335f-479d-9a1e-24590be62085-lokistack-gateway" pod "e79ac819-335f-479d-9a1e-24590be62085" (UID: "e79ac819-335f-479d-9a1e-24590be62085"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:31:36.190709 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.190678 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79ac819-335f-479d-9a1e-24590be62085-data-science-lokistack-gateway-ca-bundle" pod "e79ac819-335f-479d-9a1e-24590be62085" (UID: "e79ac819-335f-479d-9a1e-24590be62085"). InnerVolumeSpecName "data-science-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:31:36.191007 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.190982 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e79ac819-335f-479d-9a1e-24590be62085-tenants" pod "e79ac819-335f-479d-9a1e-24590be62085" (UID: "e79ac819-335f-479d-9a1e-24590be62085"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:36.215851 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.215817 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 30 11:31:36.219826 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.219803 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 30 11:31:36.233960 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.233941 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5964dd49df-kfbhf" Oct 30 11:31:36.285146 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.285117 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d6d9e7e-eec0-4132-97a0-a1a500df2de4-kube-api-access-65lwv\" (UniqueName: \"kubernetes.io/projected/5d6d9e7e-eec0-4132-97a0-a1a500df2de4-kube-api-access-65lwv\") pod \"5d6d9e7e-eec0-4132-97a0-a1a500df2de4\" (UID: \"5d6d9e7e-eec0-4132-97a0-a1a500df2de4\") " Oct 30 11:31:36.285291 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.285174 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d6d9e7e-eec0-4132-97a0-a1a500df2de4-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5d6d9e7e-eec0-4132-97a0-a1a500df2de4-otc-internal\") pod \"5d6d9e7e-eec0-4132-97a0-a1a500df2de4\" (UID: \"5d6d9e7e-eec0-4132-97a0-a1a500df2de4\") " Oct 30 11:31:36.285484 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.285464 4289 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/e79ac819-335f-479d-9a1e-24590be62085-lokistack-gateway\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:36.285525 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.285491 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gszgn\" (UniqueName: \"kubernetes.io/projected/e79ac819-335f-479d-9a1e-24590be62085-kube-api-access-gszgn\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:36.285525 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.285507 4289 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/e79ac819-335f-479d-9a1e-24590be62085-tls-secret\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:36.285525 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.285522 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/e79ac819-335f-479d-9a1e-24590be62085-data-science-lokistack-gateway-ca-bundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:36.285628 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.285524 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d6d9e7e-eec0-4132-97a0-a1a500df2de4-otc-internal" pod "5d6d9e7e-eec0-4132-97a0-a1a500df2de4" (UID: "5d6d9e7e-eec0-4132-97a0-a1a500df2de4"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:31:36.285628 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.285536 4289 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/e79ac819-335f-479d-9a1e-24590be62085-rbac\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:36.285628 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.285554 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/e79ac819-335f-479d-9a1e-24590be62085-data-science-lokistack-gateway-client-http\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:36.285628 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.285569 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e79ac819-335f-479d-9a1e-24590be62085-data-science-lokistack-ca-bundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:36.285628 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.285582 4289 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/e79ac819-335f-479d-9a1e-24590be62085-tenants\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:36.287151 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.287129 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6d9e7e-eec0-4132-97a0-a1a500df2de4-kube-api-access-65lwv" pod "5d6d9e7e-eec0-4132-97a0-a1a500df2de4" (UID: "5d6d9e7e-eec0-4132-97a0-a1a500df2de4"). InnerVolumeSpecName "kube-api-access-65lwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:31:36.386953 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.386859 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65lwv\" (UniqueName: \"kubernetes.io/projected/5d6d9e7e-eec0-4132-97a0-a1a500df2de4-kube-api-access-65lwv\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:36.386953 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.386908 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5d6d9e7e-eec0-4132-97a0-a1a500df2de4-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:36.995859 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.995813 4289 generic.go:356] "Generic (PLEG): container finished" podID="5d6d9e7e-eec0-4132-97a0-a1a500df2de4" containerID="9185d5a74dd5383d4b5e6f4962c4d2e2c23643f300d1a378a5217f04f7d707c9" exitCode=0 Oct 30 11:31:36.996316 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.995885 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5964dd49df-kfbhf" Oct 30 11:31:36.996316 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.995903 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5964dd49df-kfbhf" event={"ID":"5d6d9e7e-eec0-4132-97a0-a1a500df2de4","Type":"ContainerDied","Data":"9185d5a74dd5383d4b5e6f4962c4d2e2c23643f300d1a378a5217f04f7d707c9"} Oct 30 11:31:36.996316 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.995986 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5964dd49df-kfbhf" event={"ID":"5d6d9e7e-eec0-4132-97a0-a1a500df2de4","Type":"ContainerDied","Data":"e8f9b4be7547319783684de16a0451bf3860a35ca93c6744def4e844d8f10d83"} Oct 30 11:31:36.996316 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.996011 4289 scope.go:122] "RemoveContainer" containerID="9185d5a74dd5383d4b5e6f4962c4d2e2c23643f300d1a378a5217f04f7d707c9" Oct 30 11:31:36.999509 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.999479 4289 generic.go:356] "Generic (PLEG): container finished" podID="e79ac819-335f-479d-9a1e-24590be62085" containerID="6440099cb827dd8cc7d0fb5041abb938397aba8f09dfca988666067624faf59f" exitCode=0 Oct 30 11:31:36.999625 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.999535 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" event={"ID":"e79ac819-335f-479d-9a1e-24590be62085","Type":"ContainerDied","Data":"6440099cb827dd8cc7d0fb5041abb938397aba8f09dfca988666067624faf59f"} Oct 30 11:31:36.999625 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.999558 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" Oct 30 11:31:36.999625 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:36.999565 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x" event={"ID":"e79ac819-335f-479d-9a1e-24590be62085","Type":"ContainerDied","Data":"077533c7bb80f4a3cbd52577f47f3fe13b67925616853e782937e0c598046024"} Oct 30 11:31:37.005147 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:37.005120 4289 scope.go:122] "RemoveContainer" containerID="9185d5a74dd5383d4b5e6f4962c4d2e2c23643f300d1a378a5217f04f7d707c9" Oct 30 11:31:37.005698 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:31:37.005676 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9185d5a74dd5383d4b5e6f4962c4d2e2c23643f300d1a378a5217f04f7d707c9\": container with ID starting with 9185d5a74dd5383d4b5e6f4962c4d2e2c23643f300d1a378a5217f04f7d707c9 not found: ID does not exist" containerID="9185d5a74dd5383d4b5e6f4962c4d2e2c23643f300d1a378a5217f04f7d707c9" Oct 30 11:31:37.005786 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:37.005704 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9185d5a74dd5383d4b5e6f4962c4d2e2c23643f300d1a378a5217f04f7d707c9"} err="failed to get container status \"9185d5a74dd5383d4b5e6f4962c4d2e2c23643f300d1a378a5217f04f7d707c9\": rpc error: code = NotFound desc = could not find container \"9185d5a74dd5383d4b5e6f4962c4d2e2c23643f300d1a378a5217f04f7d707c9\": container with ID starting with 9185d5a74dd5383d4b5e6f4962c4d2e2c23643f300d1a378a5217f04f7d707c9 not found: ID does not exist" Oct 30 11:31:37.005786 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:37.005720 4289 scope.go:122] "RemoveContainer" containerID="410bc3596004029993012562e90be905e9ced78990cac3a878219725ab43701a" Oct 30 11:31:37.013710 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:37.013693 4289 scope.go:122] "RemoveContainer" containerID="6440099cb827dd8cc7d0fb5041abb938397aba8f09dfca988666067624faf59f" Oct 30 11:31:37.021088 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:37.021065 4289 scope.go:122] "RemoveContainer" containerID="410bc3596004029993012562e90be905e9ced78990cac3a878219725ab43701a" Oct 30 11:31:37.021355 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:31:37.021334 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"410bc3596004029993012562e90be905e9ced78990cac3a878219725ab43701a\": container with ID starting with 410bc3596004029993012562e90be905e9ced78990cac3a878219725ab43701a not found: ID does not exist" containerID="410bc3596004029993012562e90be905e9ced78990cac3a878219725ab43701a" Oct 30 11:31:37.021450 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:37.021360 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"410bc3596004029993012562e90be905e9ced78990cac3a878219725ab43701a"} err="failed to get container status \"410bc3596004029993012562e90be905e9ced78990cac3a878219725ab43701a\": rpc error: code = NotFound desc = could not find container \"410bc3596004029993012562e90be905e9ced78990cac3a878219725ab43701a\": container with ID starting with 410bc3596004029993012562e90be905e9ced78990cac3a878219725ab43701a not found: ID does not exist" Oct 30 11:31:37.021450 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:37.021375 4289 scope.go:122] "RemoveContainer" containerID="6440099cb827dd8cc7d0fb5041abb938397aba8f09dfca988666067624faf59f" Oct 30 11:31:37.021638 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:31:37.021621 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6440099cb827dd8cc7d0fb5041abb938397aba8f09dfca988666067624faf59f\": container with ID starting with 6440099cb827dd8cc7d0fb5041abb938397aba8f09dfca988666067624faf59f not found: ID does not exist" containerID="6440099cb827dd8cc7d0fb5041abb938397aba8f09dfca988666067624faf59f" Oct 30 11:31:37.021686 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:37.021644 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6440099cb827dd8cc7d0fb5041abb938397aba8f09dfca988666067624faf59f"} err="failed to get container status \"6440099cb827dd8cc7d0fb5041abb938397aba8f09dfca988666067624faf59f\": rpc error: code = NotFound desc = could not find container \"6440099cb827dd8cc7d0fb5041abb938397aba8f09dfca988666067624faf59f\": container with ID starting with 6440099cb827dd8cc7d0fb5041abb938397aba8f09dfca988666067624faf59f not found: ID does not exist" Oct 30 11:31:37.047502 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:37.047477 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x"] Oct 30 11:31:37.059957 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:37.059933 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-b7dfc66f6-4s62x"] Oct 30 11:31:37.086082 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:37.086059 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5964dd49df-kfbhf"] Oct 30 11:31:37.090200 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:37.090181 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5964dd49df-kfbhf"] Oct 30 11:31:37.624798 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:37.624756 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36f972dd-9d39-497e-9486-6fbc0ef2b31f" path="/var/lib/kubelet/pods/36f972dd-9d39-497e-9486-6fbc0ef2b31f/volumes" Oct 30 11:31:37.625187 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:37.625167 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d6d9e7e-eec0-4132-97a0-a1a500df2de4" path="/var/lib/kubelet/pods/5d6d9e7e-eec0-4132-97a0-a1a500df2de4/volumes" Oct 30 11:31:37.625628 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:37.625609 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d1fcc3c-d5e7-494d-8c1f-da01485a222d" path="/var/lib/kubelet/pods/8d1fcc3c-d5e7-494d-8c1f-da01485a222d/volumes" Oct 30 11:31:37.626310 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:37.626290 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e79ac819-335f-479d-9a1e-24590be62085" path="/var/lib/kubelet/pods/e79ac819-335f-479d-9a1e-24590be62085/volumes" Oct 30 11:31:37.626778 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:37.626761 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eebd0be1-9d9a-4524-8554-9e479a10b97a" path="/var/lib/kubelet/pods/eebd0be1-9d9a-4524-8554-9e479a10b97a/volumes" Oct 30 11:31:39.623150 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:39.623126 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" Oct 30 11:31:39.716328 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:39.716250 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23be38ee-7767-4a92-a7cf-bf84b3690494-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23be38ee-7767-4a92-a7cf-bf84b3690494-data-science-lokistack-ca-bundle\") pod \"23be38ee-7767-4a92-a7cf-bf84b3690494\" (UID: \"23be38ee-7767-4a92-a7cf-bf84b3690494\") " Oct 30 11:31:39.716328 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:39.716290 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23be38ee-7767-4a92-a7cf-bf84b3690494-data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/23be38ee-7767-4a92-a7cf-bf84b3690494-data-science-lokistack-distributor-grpc\") pod \"23be38ee-7767-4a92-a7cf-bf84b3690494\" (UID: \"23be38ee-7767-4a92-a7cf-bf84b3690494\") " Oct 30 11:31:39.716328 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:39.716324 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23be38ee-7767-4a92-a7cf-bf84b3690494-kube-api-access-g8x7p\" (UniqueName: \"kubernetes.io/projected/23be38ee-7767-4a92-a7cf-bf84b3690494-kube-api-access-g8x7p\") pod \"23be38ee-7767-4a92-a7cf-bf84b3690494\" (UID: \"23be38ee-7767-4a92-a7cf-bf84b3690494\") " Oct 30 11:31:39.716602 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:39.716366 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23be38ee-7767-4a92-a7cf-bf84b3690494-config\" (UniqueName: \"kubernetes.io/configmap/23be38ee-7767-4a92-a7cf-bf84b3690494-config\") pod \"23be38ee-7767-4a92-a7cf-bf84b3690494\" (UID: \"23be38ee-7767-4a92-a7cf-bf84b3690494\") " Oct 30 11:31:39.716602 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:39.716383 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23be38ee-7767-4a92-a7cf-bf84b3690494-data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/23be38ee-7767-4a92-a7cf-bf84b3690494-data-science-lokistack-distributor-http\") pod \"23be38ee-7767-4a92-a7cf-bf84b3690494\" (UID: \"23be38ee-7767-4a92-a7cf-bf84b3690494\") " Oct 30 11:31:39.716754 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:39.716726 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23be38ee-7767-4a92-a7cf-bf84b3690494-data-science-lokistack-ca-bundle" pod "23be38ee-7767-4a92-a7cf-bf84b3690494" (UID: "23be38ee-7767-4a92-a7cf-bf84b3690494"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:31:39.716860 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:39.716835 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23be38ee-7767-4a92-a7cf-bf84b3690494-config" pod "23be38ee-7767-4a92-a7cf-bf84b3690494" (UID: "23be38ee-7767-4a92-a7cf-bf84b3690494"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:31:39.718524 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:39.718497 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23be38ee-7767-4a92-a7cf-bf84b3690494-data-science-lokistack-distributor-http" pod "23be38ee-7767-4a92-a7cf-bf84b3690494" (UID: "23be38ee-7767-4a92-a7cf-bf84b3690494"). InnerVolumeSpecName "data-science-lokistack-distributor-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:39.718632 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:39.718530 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23be38ee-7767-4a92-a7cf-bf84b3690494-data-science-lokistack-distributor-grpc" pod "23be38ee-7767-4a92-a7cf-bf84b3690494" (UID: "23be38ee-7767-4a92-a7cf-bf84b3690494"). InnerVolumeSpecName "data-science-lokistack-distributor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:31:39.718632 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:39.718544 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23be38ee-7767-4a92-a7cf-bf84b3690494-kube-api-access-g8x7p" pod "23be38ee-7767-4a92-a7cf-bf84b3690494" (UID: "23be38ee-7767-4a92-a7cf-bf84b3690494"). InnerVolumeSpecName "kube-api-access-g8x7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:31:39.817765 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:39.817713 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23be38ee-7767-4a92-a7cf-bf84b3690494-data-science-lokistack-ca-bundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:39.817765 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:39.817760 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/23be38ee-7767-4a92-a7cf-bf84b3690494-data-science-lokistack-distributor-grpc\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:39.817765 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:39.817771 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8x7p\" (UniqueName: \"kubernetes.io/projected/23be38ee-7767-4a92-a7cf-bf84b3690494-kube-api-access-g8x7p\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:39.817765 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:39.817781 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23be38ee-7767-4a92-a7cf-bf84b3690494-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:39.818047 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:39.817790 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/23be38ee-7767-4a92-a7cf-bf84b3690494-data-science-lokistack-distributor-http\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:31:40.015884 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:40.015850 4289 generic.go:356] "Generic (PLEG): container finished" podID="23be38ee-7767-4a92-a7cf-bf84b3690494" containerID="28653428c5505bf3de820dfd9d6fe81f2d4088b6cc0079f005a0a3e992925380" exitCode=0 Oct 30 11:31:40.016049 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:40.015900 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" Oct 30 11:31:40.016049 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:40.015918 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" event={"ID":"23be38ee-7767-4a92-a7cf-bf84b3690494","Type":"ContainerDied","Data":"28653428c5505bf3de820dfd9d6fe81f2d4088b6cc0079f005a0a3e992925380"} Oct 30 11:31:40.016049 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:40.015957 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf" event={"ID":"23be38ee-7767-4a92-a7cf-bf84b3690494","Type":"ContainerDied","Data":"f85567f22d13173d2c468233b08760ad12288154917fd701db59a919cdce68db"} Oct 30 11:31:40.016049 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:40.015975 4289 scope.go:122] "RemoveContainer" containerID="28653428c5505bf3de820dfd9d6fe81f2d4088b6cc0079f005a0a3e992925380" Oct 30 11:31:40.025037 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:40.025020 4289 scope.go:122] "RemoveContainer" containerID="28653428c5505bf3de820dfd9d6fe81f2d4088b6cc0079f005a0a3e992925380" Oct 30 11:31:40.025279 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:31:40.025262 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28653428c5505bf3de820dfd9d6fe81f2d4088b6cc0079f005a0a3e992925380\": container with ID starting with 28653428c5505bf3de820dfd9d6fe81f2d4088b6cc0079f005a0a3e992925380 not found: ID does not exist" containerID="28653428c5505bf3de820dfd9d6fe81f2d4088b6cc0079f005a0a3e992925380" Oct 30 11:31:40.025327 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:40.025288 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28653428c5505bf3de820dfd9d6fe81f2d4088b6cc0079f005a0a3e992925380"} err="failed to get container status \"28653428c5505bf3de820dfd9d6fe81f2d4088b6cc0079f005a0a3e992925380\": rpc error: code = NotFound desc = could not find container \"28653428c5505bf3de820dfd9d6fe81f2d4088b6cc0079f005a0a3e992925380\": container with ID starting with 28653428c5505bf3de820dfd9d6fe81f2d4088b6cc0079f005a0a3e992925380 not found: ID does not exist" Oct 30 11:31:40.078922 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:40.078892 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf"] Oct 30 11:31:40.090674 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:40.090647 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-dtlmf"] Oct 30 11:31:41.625037 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:41.625005 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23be38ee-7767-4a92-a7cf-bf84b3690494" path="/var/lib/kubelet/pods/23be38ee-7767-4a92-a7cf-bf84b3690494/volumes" Oct 30 11:31:56.257133 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:31:56.257083 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:31:58.870843 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:31:58.870804 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:05.091222 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:05.091186 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:07.173855 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.173816 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:07.348737 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.348702 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:07.477543 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.477511 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl"] Oct 30 11:32:07.477850 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.477837 4289 cpu_manager.go:414] "containerMap: removing container" podUID="e79ac819-335f-479d-9a1e-24590be62085" containerName="gateway" Oct 30 11:32:07.477899 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.477851 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e79ac819-335f-479d-9a1e-24590be62085" containerName="gateway" Oct 30 11:32:07.477899 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.477861 4289 cpu_manager.go:414] "containerMap: removing container" podUID="eebd0be1-9d9a-4524-8554-9e479a10b97a" containerName="loki-compactor" Oct 30 11:32:07.477899 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.477866 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eebd0be1-9d9a-4524-8554-9e479a10b97a" containerName="loki-compactor" Oct 30 11:32:07.477899 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.477883 4289 cpu_manager.go:414] "containerMap: removing container" podUID="5d6d9e7e-eec0-4132-97a0-a1a500df2de4" containerName="otc-container" Oct 30 11:32:07.477899 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.477889 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6d9e7e-eec0-4132-97a0-a1a500df2de4" containerName="otc-container" Oct 30 11:32:07.477899 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.477896 4289 cpu_manager.go:414] "containerMap: removing container" podUID="23be38ee-7767-4a92-a7cf-bf84b3690494" containerName="loki-distributor" Oct 30 11:32:07.477899 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.477900 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23be38ee-7767-4a92-a7cf-bf84b3690494" containerName="loki-distributor" Oct 30 11:32:07.478116 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.477906 4289 cpu_manager.go:414] "containerMap: removing container" podUID="36f972dd-9d39-497e-9486-6fbc0ef2b31f" containerName="loki-index-gateway" Oct 30 11:32:07.478116 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.477910 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36f972dd-9d39-497e-9486-6fbc0ef2b31f" containerName="loki-index-gateway" Oct 30 11:32:07.478116 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.477917 4289 cpu_manager.go:414] "containerMap: removing container" podUID="e79ac819-335f-479d-9a1e-24590be62085" containerName="opa" Oct 30 11:32:07.478116 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.477922 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e79ac819-335f-479d-9a1e-24590be62085" containerName="opa" Oct 30 11:32:07.478116 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.477928 4289 cpu_manager.go:414] "containerMap: removing container" podUID="8d1fcc3c-d5e7-494d-8c1f-da01485a222d" containerName="loki-ingester" Oct 30 11:32:07.478116 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.477932 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d1fcc3c-d5e7-494d-8c1f-da01485a222d" containerName="loki-ingester" Oct 30 11:32:07.478116 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.477987 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="e79ac819-335f-479d-9a1e-24590be62085" containerName="gateway" Oct 30 11:32:07.478116 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.477994 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="eebd0be1-9d9a-4524-8554-9e479a10b97a" containerName="loki-compactor" Oct 30 11:32:07.478116 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.478002 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d6d9e7e-eec0-4132-97a0-a1a500df2de4" containerName="otc-container" Oct 30 11:32:07.478116 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.478007 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="23be38ee-7767-4a92-a7cf-bf84b3690494" containerName="loki-distributor" Oct 30 11:32:07.478116 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.478013 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="36f972dd-9d39-497e-9486-6fbc0ef2b31f" containerName="loki-index-gateway" Oct 30 11:32:07.478116 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.478018 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="e79ac819-335f-479d-9a1e-24590be62085" containerName="opa" Oct 30 11:32:07.478116 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.478023 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d1fcc3c-d5e7-494d-8c1f-da01485a222d" containerName="loki-ingester" Oct 30 11:32:07.482219 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.482203 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" Oct 30 11:32:07.514569 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.514531 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 11:32:07.514569 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.514531 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 11:32:07.514832 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.514808 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 11:32:07.514832 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.514812 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 11:32:07.515101 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.514846 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 11:32:07.515101 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.515074 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-xq7g2\"" Oct 30 11:32:07.515236 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.515220 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 11:32:07.516003 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.515980 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl"] Oct 30 11:32:07.528316 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.528294 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl"] Oct 30 11:32:07.532004 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.531976 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" Oct 30 11:32:07.533252 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.533230 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:07.545061 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.545039 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl"] Oct 30 11:32:07.566007 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.565972 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w"] Oct 30 11:32:07.569672 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.569653 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" Oct 30 11:32:07.590071 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.590046 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w"] Oct 30 11:32:07.602225 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.602202 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 11:32:07.602466 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.602418 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 11:32:07.602592 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.602460 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 11:32:07.602592 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.602469 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-qrl6p\"" Oct 30 11:32:07.602592 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.602552 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 11:32:07.602767 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.602589 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-zvjld\"" Oct 30 11:32:07.602767 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.602637 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 11:32:07.604476 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.604457 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r"] Oct 30 11:32:07.607809 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.607792 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r" Oct 30 11:32:07.627062 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.627034 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r"] Oct 30 11:32:07.649815 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.649779 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/953398bd-24a9-4221-88bd-601a0796cf64-client-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl\" (UID: \"953398bd-24a9-4221-88bd-601a0796cf64\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" Oct 30 11:32:07.649983 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.649839 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2qlc\" (UniqueName: \"kubernetes.io/projected/62897926-e2e9-4408-b407-b1bdeb4ee0e9-kube-api-access-s2qlc\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nrqbl\" (UID: \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" Oct 30 11:32:07.649983 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.649937 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e9599347-1676-48b3-bf20-5ba203487718-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-56n4w\" (UID: \"e9599347-1676-48b3-bf20-5ba203487718\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" Oct 30 11:32:07.649983 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.649977 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e9599347-1676-48b3-bf20-5ba203487718-config\") pod \"data-science-thanos-querier-proxy-7b56895dd5-56n4w\" (UID: \"e9599347-1676-48b3-bf20-5ba203487718\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" Oct 30 11:32:07.650117 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.650010 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/62897926-e2e9-4408-b407-b1bdeb4ee0e9-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nrqbl\" (UID: \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" Oct 30 11:32:07.650158 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.650113 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/953398bd-24a9-4221-88bd-601a0796cf64-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl\" (UID: \"953398bd-24a9-4221-88bd-601a0796cf64\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" Oct 30 11:32:07.650192 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.650149 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/953398bd-24a9-4221-88bd-601a0796cf64-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl\" (UID: \"953398bd-24a9-4221-88bd-601a0796cf64\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" Oct 30 11:32:07.650280 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.650263 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nrqbl\" (UID: \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" Oct 30 11:32:07.650333 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.650317 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nrqbl\" (UID: \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" Oct 30 11:32:07.650371 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.650349 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v97rg\" (UniqueName: \"kubernetes.io/projected/e9599347-1676-48b3-bf20-5ba203487718-kube-api-access-v97rg\") pod \"data-science-thanos-querier-proxy-7b56895dd5-56n4w\" (UID: \"e9599347-1676-48b3-bf20-5ba203487718\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" Oct 30 11:32:07.650412 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.650394 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/953398bd-24a9-4221-88bd-601a0796cf64-config\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl\" (UID: \"953398bd-24a9-4221-88bd-601a0796cf64\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" Oct 30 11:32:07.650524 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.650505 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nrqbl\" (UID: \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" Oct 30 11:32:07.650564 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.650542 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz52z\" (UniqueName: \"kubernetes.io/projected/953398bd-24a9-4221-88bd-601a0796cf64-kube-api-access-pz52z\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl\" (UID: \"953398bd-24a9-4221-88bd-601a0796cf64\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" Oct 30 11:32:07.696423 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.696387 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:07.703128 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.703111 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-lr8vt\"" Oct 30 11:32:07.703228 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.703184 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 11:32:07.703267 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.703244 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 11:32:07.751550 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.751482 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2qlc\" (UniqueName: \"kubernetes.io/projected/62897926-e2e9-4408-b407-b1bdeb4ee0e9-kube-api-access-s2qlc\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nrqbl\" (UID: \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" Oct 30 11:32:07.751550 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.751515 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ade4e9f0-3694-4382-a2e9-5105763562b6-config\") pod \"korrel8r-5cbcc89b8f-b5v4r\" (UID: \"ade4e9f0-3694-4382-a2e9-5105763562b6\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r" Oct 30 11:32:07.751550 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.751532 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ade4e9f0-3694-4382-a2e9-5105763562b6-tls\") pod \"korrel8r-5cbcc89b8f-b5v4r\" (UID: \"ade4e9f0-3694-4382-a2e9-5105763562b6\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r" Oct 30 11:32:07.751727 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.751609 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e9599347-1676-48b3-bf20-5ba203487718-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-56n4w\" (UID: \"e9599347-1676-48b3-bf20-5ba203487718\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" Oct 30 11:32:07.751727 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.751645 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e9599347-1676-48b3-bf20-5ba203487718-config\") pod \"data-science-thanos-querier-proxy-7b56895dd5-56n4w\" (UID: \"e9599347-1676-48b3-bf20-5ba203487718\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" Oct 30 11:32:07.751727 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.751668 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/62897926-e2e9-4408-b407-b1bdeb4ee0e9-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nrqbl\" (UID: \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" Oct 30 11:32:07.751727 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.751697 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/953398bd-24a9-4221-88bd-601a0796cf64-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl\" (UID: \"953398bd-24a9-4221-88bd-601a0796cf64\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" Oct 30 11:32:07.751945 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.751837 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/953398bd-24a9-4221-88bd-601a0796cf64-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl\" (UID: \"953398bd-24a9-4221-88bd-601a0796cf64\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" Oct 30 11:32:07.751945 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.751938 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nrqbl\" (UID: \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" Oct 30 11:32:07.752051 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.751974 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nrqbl\" (UID: \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" Oct 30 11:32:07.752109 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:32:07.752058 4289 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-prometheus-cluster-proxy-tls: secret "data-science-prometheus-cluster-proxy-tls" not found Oct 30 11:32:07.752160 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:32:07.752150 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-tls podName:62897926-e2e9-4408-b407-b1bdeb4ee0e9 nodeName:}" failed. No retries permitted until 2026-10-30 11:32:08.252126 +0000 UTC m=+2818.753505483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-tls") pod "data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" (UID: "62897926-e2e9-4408-b407-b1bdeb4ee0e9") : secret "data-science-prometheus-cluster-proxy-tls" not found Oct 30 11:32:07.752238 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:32:07.752196 4289 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 30 11:32:07.752289 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:32:07.752279 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-prometheus-client-cert podName:62897926-e2e9-4408-b407-b1bdeb4ee0e9 nodeName:}" failed. No retries permitted until 2026-10-30 11:32:08.252261512 +0000 UTC m=+2818.753640990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" (UID: "62897926-e2e9-4408-b407-b1bdeb4ee0e9") : secret "prometheus-operated-tls" not found Oct 30 11:32:07.752345 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.752304 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v97rg\" (UniqueName: \"kubernetes.io/projected/e9599347-1676-48b3-bf20-5ba203487718-kube-api-access-v97rg\") pod \"data-science-thanos-querier-proxy-7b56895dd5-56n4w\" (UID: \"e9599347-1676-48b3-bf20-5ba203487718\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" Oct 30 11:32:07.752345 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.752334 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/953398bd-24a9-4221-88bd-601a0796cf64-config\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl\" (UID: \"953398bd-24a9-4221-88bd-601a0796cf64\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" Oct 30 11:32:07.752480 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.752463 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nrqbl\" (UID: \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" Oct 30 11:32:07.752548 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.752486 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e9599347-1676-48b3-bf20-5ba203487718-config\") pod \"data-science-thanos-querier-proxy-7b56895dd5-56n4w\" (UID: \"e9599347-1676-48b3-bf20-5ba203487718\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" Oct 30 11:32:07.752548 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.752495 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcz7g\" (UniqueName: \"kubernetes.io/projected/ade4e9f0-3694-4382-a2e9-5105763562b6-kube-api-access-mcz7g\") pod \"korrel8r-5cbcc89b8f-b5v4r\" (UID: \"ade4e9f0-3694-4382-a2e9-5105763562b6\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r" Oct 30 11:32:07.752548 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.752516 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/953398bd-24a9-4221-88bd-601a0796cf64-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl\" (UID: \"953398bd-24a9-4221-88bd-601a0796cf64\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" Oct 30 11:32:07.752548 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.752536 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz52z\" (UniqueName: \"kubernetes.io/projected/953398bd-24a9-4221-88bd-601a0796cf64-kube-api-access-pz52z\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl\" (UID: \"953398bd-24a9-4221-88bd-601a0796cf64\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" Oct 30 11:32:07.752770 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.752571 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/953398bd-24a9-4221-88bd-601a0796cf64-client-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl\" (UID: \"953398bd-24a9-4221-88bd-601a0796cf64\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" Oct 30 11:32:07.752999 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.752969 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/953398bd-24a9-4221-88bd-601a0796cf64-config\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl\" (UID: \"953398bd-24a9-4221-88bd-601a0796cf64\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" Oct 30 11:32:07.753161 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.753138 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/953398bd-24a9-4221-88bd-601a0796cf64-client-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl\" (UID: \"953398bd-24a9-4221-88bd-601a0796cf64\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" Oct 30 11:32:07.753392 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.753373 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/62897926-e2e9-4408-b407-b1bdeb4ee0e9-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nrqbl\" (UID: \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" Oct 30 11:32:07.754485 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.754468 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e9599347-1676-48b3-bf20-5ba203487718-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-56n4w\" (UID: \"e9599347-1676-48b3-bf20-5ba203487718\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" Oct 30 11:32:07.754573 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.754547 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/953398bd-24a9-4221-88bd-601a0796cf64-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl\" (UID: \"953398bd-24a9-4221-88bd-601a0796cf64\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" Oct 30 11:32:07.754930 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.754911 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nrqbl\" (UID: \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" Oct 30 11:32:07.774693 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.774668 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2qlc\" (UniqueName: \"kubernetes.io/projected/62897926-e2e9-4408-b407-b1bdeb4ee0e9-kube-api-access-s2qlc\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nrqbl\" (UID: \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" Oct 30 11:32:07.776108 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.776083 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz52z\" (UniqueName: \"kubernetes.io/projected/953398bd-24a9-4221-88bd-601a0796cf64-kube-api-access-pz52z\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl\" (UID: \"953398bd-24a9-4221-88bd-601a0796cf64\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" Oct 30 11:32:07.776642 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.776618 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v97rg\" (UniqueName: \"kubernetes.io/projected/e9599347-1676-48b3-bf20-5ba203487718-kube-api-access-v97rg\") pod \"data-science-thanos-querier-proxy-7b56895dd5-56n4w\" (UID: \"e9599347-1676-48b3-bf20-5ba203487718\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" Oct 30 11:32:07.849523 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.849489 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" Oct 30 11:32:07.854020 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.853644 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcz7g\" (UniqueName: \"kubernetes.io/projected/ade4e9f0-3694-4382-a2e9-5105763562b6-kube-api-access-mcz7g\") pod \"korrel8r-5cbcc89b8f-b5v4r\" (UID: \"ade4e9f0-3694-4382-a2e9-5105763562b6\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r" Oct 30 11:32:07.854020 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.853703 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ade4e9f0-3694-4382-a2e9-5105763562b6-config\") pod \"korrel8r-5cbcc89b8f-b5v4r\" (UID: \"ade4e9f0-3694-4382-a2e9-5105763562b6\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r" Oct 30 11:32:07.854020 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.853726 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ade4e9f0-3694-4382-a2e9-5105763562b6-tls\") pod \"korrel8r-5cbcc89b8f-b5v4r\" (UID: \"ade4e9f0-3694-4382-a2e9-5105763562b6\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r" Oct 30 11:32:07.854508 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.854485 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ade4e9f0-3694-4382-a2e9-5105763562b6-config\") pod \"korrel8r-5cbcc89b8f-b5v4r\" (UID: \"ade4e9f0-3694-4382-a2e9-5105763562b6\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r" Oct 30 11:32:07.856371 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.856346 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ade4e9f0-3694-4382-a2e9-5105763562b6-tls\") pod \"korrel8r-5cbcc89b8f-b5v4r\" (UID: \"ade4e9f0-3694-4382-a2e9-5105763562b6\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r" Oct 30 11:32:07.877360 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.877336 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcz7g\" (UniqueName: \"kubernetes.io/projected/ade4e9f0-3694-4382-a2e9-5105763562b6-kube-api-access-mcz7g\") pod \"korrel8r-5cbcc89b8f-b5v4r\" (UID: \"ade4e9f0-3694-4382-a2e9-5105763562b6\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r" Oct 30 11:32:07.887106 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.887079 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" Oct 30 11:32:07.920160 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:07.920129 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r" Oct 30 11:32:08.148026 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:08.148000 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl"] Oct 30 11:32:08.149616 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:32:08.149588 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod953398bd_24a9_4221_88bd_601a0796cf64.slice/crio-83bf9c8b4b995c12966f113f39c5ee125af7743a9d3ef8354f4f062c511eabb1 WatchSource:0}: Error finding container 83bf9c8b4b995c12966f113f39c5ee125af7743a9d3ef8354f4f062c511eabb1: Status 404 returned error can't find the container with id 83bf9c8b4b995c12966f113f39c5ee125af7743a9d3ef8354f4f062c511eabb1 Oct 30 11:32:08.199277 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:08.199252 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w"] Oct 30 11:32:08.200730 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:08.200698 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:08.201038 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:32:08.201017 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9599347_1676_48b3_bf20_5ba203487718.slice/crio-3aab1c3b3637a463594ff1bfcfbb3eef5180769ccff2cec008adc0b194cf7f2f WatchSource:0}: Error finding container 3aab1c3b3637a463594ff1bfcfbb3eef5180769ccff2cec008adc0b194cf7f2f: Status 404 returned error can't find the container with id 3aab1c3b3637a463594ff1bfcfbb3eef5180769ccff2cec008adc0b194cf7f2f Oct 30 11:32:08.225533 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:08.225500 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r"] Oct 30 11:32:08.238350 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:32:08.238328 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade4e9f0_3694_4382_a2e9_5105763562b6.slice/crio-138684da53fc62550f29a0491425d9e6c94a392bf20701321c090601f9695c19 WatchSource:0}: Error finding container 138684da53fc62550f29a0491425d9e6c94a392bf20701321c090601f9695c19: Status 404 returned error can't find the container with id 138684da53fc62550f29a0491425d9e6c94a392bf20701321c090601f9695c19 Oct 30 11:32:08.257374 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:08.257351 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nrqbl\" (UID: \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" Oct 30 11:32:08.257490 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:08.257384 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nrqbl\" (UID: \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" Oct 30 11:32:08.259639 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:08.259617 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nrqbl\" (UID: \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" Oct 30 11:32:08.259787 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:08.259766 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nrqbl\" (UID: \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" Oct 30 11:32:08.383816 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:08.383785 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:08.395165 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:08.395144 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" Oct 30 11:32:08.555653 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:08.555611 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:08.694089 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:08.694058 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl"] Oct 30 11:32:08.695820 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:32:08.695792 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62897926_e2e9_4408_b407_b1bdeb4ee0e9.slice/crio-efb8d4074b533e97225b3b2fe3f87cd9ccc7832ee8d3682e5a6452f5ac3f234f WatchSource:0}: Error finding container efb8d4074b533e97225b3b2fe3f87cd9ccc7832ee8d3682e5a6452f5ac3f234f: Status 404 returned error can't find the container with id efb8d4074b533e97225b3b2fe3f87cd9ccc7832ee8d3682e5a6452f5ac3f234f Oct 30 11:32:09.141015 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:09.140972 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" event={"ID":"e9599347-1676-48b3-bf20-5ba203487718","Type":"ContainerStarted","Data":"845c046f53f477aadd9728ceecd8603c1c4b9dc74ef60ccd2fd84b83a79731fa"} Oct 30 11:32:09.141194 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:09.141061 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" event={"ID":"e9599347-1676-48b3-bf20-5ba203487718","Type":"ContainerStarted","Data":"7e321e52776566c38897e08cefdc4a28f01959033fed400e7fb51fa4d1c3b125"} Oct 30 11:32:09.141194 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:09.141107 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" event={"ID":"e9599347-1676-48b3-bf20-5ba203487718","Type":"ContainerStarted","Data":"09347ec76edbb0b2c9acf02642b8e413dd84ef9bc7cd47461b8ed747cf3499f4"} Oct 30 11:32:09.141194 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:09.141151 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" event={"ID":"e9599347-1676-48b3-bf20-5ba203487718","Type":"ContainerStarted","Data":"3aab1c3b3637a463594ff1bfcfbb3eef5180769ccff2cec008adc0b194cf7f2f"} Oct 30 11:32:09.143187 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:09.143161 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" event={"ID":"953398bd-24a9-4221-88bd-601a0796cf64","Type":"ContainerStarted","Data":"78520c074aa268c8db0a53ee76495756244a44435d3c7800ed40c4de29f8baf7"} Oct 30 11:32:09.143317 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:09.143225 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" event={"ID":"953398bd-24a9-4221-88bd-601a0796cf64","Type":"ContainerStarted","Data":"f1837b2cdcc2402f8b98c1b33ec1d0f624b2024942a1964e405d945480d13140"} Oct 30 11:32:09.143317 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:09.143264 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" event={"ID":"953398bd-24a9-4221-88bd-601a0796cf64","Type":"ContainerStarted","Data":"0cbabac544cdd981b0e7d1042b9f8cef758eb32e37c0b4475a62b406557596dd"} Oct 30 11:32:09.143317 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:09.143297 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" event={"ID":"953398bd-24a9-4221-88bd-601a0796cf64","Type":"ContainerStarted","Data":"83bf9c8b4b995c12966f113f39c5ee125af7743a9d3ef8354f4f062c511eabb1"} Oct 30 11:32:09.144450 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:09.144407 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" event={"ID":"62897926-e2e9-4408-b407-b1bdeb4ee0e9","Type":"ContainerStarted","Data":"efb8d4074b533e97225b3b2fe3f87cd9ccc7832ee8d3682e5a6452f5ac3f234f"} Oct 30 11:32:09.145836 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:09.145810 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r" event={"ID":"ade4e9f0-3694-4382-a2e9-5105763562b6","Type":"ContainerStarted","Data":"474b33182b99ee42a1fd1b4fa4974661aa72552251b840386f3136ddb3dbba03"} Oct 30 11:32:09.145924 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:09.145848 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r" event={"ID":"ade4e9f0-3694-4382-a2e9-5105763562b6","Type":"ContainerStarted","Data":"138684da53fc62550f29a0491425d9e6c94a392bf20701321c090601f9695c19"} Oct 30 11:32:09.145924 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:09.145868 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r" Oct 30 11:32:09.179667 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:09.179633 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:09.191845 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:09.191796 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" podStartSLOduration=2.191784382 podStartE2EDuration="2.191784382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:32:09.191231484 +0000 UTC m=+2819.692610969" watchObservedRunningTime="2026-10-30 11:32:09.191784382 +0000 UTC m=+2819.693163864" Oct 30 11:32:09.245577 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:09.245509 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r" podStartSLOduration=2.245491132 podStartE2EDuration="2.245491132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:32:09.244236074 +0000 UTC m=+2819.745615561" watchObservedRunningTime="2026-10-30 11:32:09.245491132 +0000 UTC m=+2819.746870630" Oct 30 11:32:09.294717 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:09.294669 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" podStartSLOduration=2.2946555330000002 podStartE2EDuration="2.294655533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:32:09.293943746 +0000 UTC m=+2819.795323245" watchObservedRunningTime="2026-10-30 11:32:09.294655533 +0000 UTC m=+2819.796035017" Oct 30 11:32:09.357579 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:09.357505 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:10.155259 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:10.155220 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" event={"ID":"62897926-e2e9-4408-b407-b1bdeb4ee0e9","Type":"ContainerStarted","Data":"dc31cad60c6be0b68da3b6460c0d2ebc0dfff678700947175123b025d1d1a6d2"} Oct 30 11:32:10.155886 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:10.155864 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" Oct 30 11:32:10.199557 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:10.199506 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" podStartSLOduration=3.199493054 podStartE2EDuration="3.199493054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:32:10.198764072 +0000 UTC m=+2820.700143568" watchObservedRunningTime="2026-10-30 11:32:10.199493054 +0000 UTC m=+2820.700872539" Oct 30 11:32:11.796219 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:11.796174 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:12.035085 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.035053 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:12.198007 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.197912 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:12.375488 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.375449 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:32:12.381663 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.381638 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:12.407311 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.407277 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:12.410725 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.410699 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:32:12.480914 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.480884 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 11:32:12.481165 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.480933 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 11:32:12.481165 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.480977 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 11:32:12.481165 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.481049 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 11:32:12.481165 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.481091 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-vp8p7\"" Oct 30 11:32:12.492392 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.492367 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:12.492568 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.492402 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:12.492568 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.492558 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb258\" (UniqueName: \"kubernetes.io/projected/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-kube-api-access-bb258\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:12.492721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.492685 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:12.492900 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.492869 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:12.492900 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.492900 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:12.493050 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.492919 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:12.515167 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.515138 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:32:12.518842 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.518827 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.548051 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.548017 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:32:12.575443 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.575395 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:12.579824 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.579798 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 11:32:12.579987 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.579866 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 11:32:12.579987 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.579908 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 11:32:12.579987 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.579914 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 11:32:12.579987 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.579920 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 11:32:12.580127 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.580077 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-2lb6p\"" Oct 30 11:32:12.581295 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.581282 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 11:32:12.581359 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.581304 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 11:32:12.581590 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.581575 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 11:32:12.594002 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.593976 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:12.594110 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.594025 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:12.594110 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.594049 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:12.594110 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.594073 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:12.594270 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.594135 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:12.594270 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.594164 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:12.594270 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.594197 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb258\" (UniqueName: \"kubernetes.io/projected/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-kube-api-access-bb258\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:12.594767 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.594739 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:12.596687 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.596666 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:12.596794 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.596696 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:12.596794 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.596789 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:12.596913 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.596886 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:12.597305 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.597285 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:12.617558 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.617526 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb258\" (UniqueName: \"kubernetes.io/projected/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-kube-api-access-bb258\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:12.695648 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.695608 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cfa1bffd-a96d-493b-89d8-76406e3e054d-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.695648 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.695650 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cfa1bffd-a96d-493b-89d8-76406e3e054d-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.695905 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.695675 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cfa1bffd-a96d-493b-89d8-76406e3e054d-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.695905 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.695789 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9tnh\" (UniqueName: \"kubernetes.io/projected/cfa1bffd-a96d-493b-89d8-76406e3e054d-kube-api-access-f9tnh\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.695905 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.695880 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.696073 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.695911 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.696073 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.695964 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.696073 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.696030 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.696073 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.696060 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.696272 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.696079 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cfa1bffd-a96d-493b-89d8-76406e3e054d-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.696272 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.696102 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cfa1bffd-a96d-493b-89d8-76406e3e054d-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.696272 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.696178 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.699504 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.699486 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:12.797001 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.796965 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cfa1bffd-a96d-493b-89d8-76406e3e054d-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.797001 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.797005 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cfa1bffd-a96d-493b-89d8-76406e3e054d-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.797556 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.797023 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cfa1bffd-a96d-493b-89d8-76406e3e054d-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.797556 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.797085 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9tnh\" (UniqueName: \"kubernetes.io/projected/cfa1bffd-a96d-493b-89d8-76406e3e054d-kube-api-access-f9tnh\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.797556 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.797121 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.797556 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.797147 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.797556 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.797190 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.797556 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.797221 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.797556 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.797266 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.797556 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.797287 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cfa1bffd-a96d-493b-89d8-76406e3e054d-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.797556 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.797313 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cfa1bffd-a96d-493b-89d8-76406e3e054d-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.797556 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.797341 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.798088 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.797688 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cfa1bffd-a96d-493b-89d8-76406e3e054d-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.798253 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.798224 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cfa1bffd-a96d-493b-89d8-76406e3e054d-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.798332 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.798267 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cfa1bffd-a96d-493b-89d8-76406e3e054d-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.800275 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.800224 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cfa1bffd-a96d-493b-89d8-76406e3e054d-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.800452 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.800411 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.800554 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.800523 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.800617 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.800596 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.800862 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.800842 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cfa1bffd-a96d-493b-89d8-76406e3e054d-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.800910 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.800892 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.801029 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.801011 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.802086 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.802069 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:12.820106 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:12.820071 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9tnh\" (UniqueName: \"kubernetes.io/projected/cfa1bffd-a96d-493b-89d8-76406e3e054d-kube-api-access-f9tnh\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:13.012569 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:13.012487 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:32:13.015196 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:32:13.015166 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca51d32e_4be1_4a5a_b58f_c1fcd84cf86e.slice/crio-5496d82bcc77e68cf1107846c89a90041b607575f0eddb74ff49d7e8adb89f73 WatchSource:0}: Error finding container 5496d82bcc77e68cf1107846c89a90041b607575f0eddb74ff49d7e8adb89f73: Status 404 returned error can't find the container with id 5496d82bcc77e68cf1107846c89a90041b607575f0eddb74ff49d7e8adb89f73 Oct 30 11:32:13.170327 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:13.170292 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e","Type":"ContainerStarted","Data":"5496d82bcc77e68cf1107846c89a90041b607575f0eddb74ff49d7e8adb89f73"} Oct 30 11:32:14.176042 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:14.176005 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e","Type":"ContainerStarted","Data":"4fabd20cba075c0d4650b9abdaea12702060682fa315f9d440d76709d16afba9"} Oct 30 11:32:15.018406 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:15.018370 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:15.047705 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:15.047674 4289 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:15.118857 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:15.118819 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:15.119001 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:15.118962 4289 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:15.133293 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:15.133264 4289 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") DevicePath \"csi-ea06c66241a0a003a5daff5d9fc602f40e53768b00b104c5e4118b92d558c090\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:15.234181 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:15.234152 4289 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7342da0537aa528cc1545d2018ac3fd0382dfe4f670067e58cf8ba34bb891174/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:15.261354 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:15.261327 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:15.539188 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:15.539151 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:15.870010 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:32:15.869981 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa1bffd_a96d_493b_89d8_76406e3e054d.slice/crio-ed50e7024e314c7c9b2738185e487045374a6c3a6bcb85e18690ecd5fbf9fd49 WatchSource:0}: Error finding container ed50e7024e314c7c9b2738185e487045374a6c3a6bcb85e18690ecd5fbf9fd49: Status 404 returned error can't find the container with id ed50e7024e314c7c9b2738185e487045374a6c3a6bcb85e18690ecd5fbf9fd49 Oct 30 11:32:15.881146 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:15.881119 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:32:16.186198 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:16.186113 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"cfa1bffd-a96d-493b-89d8-76406e3e054d","Type":"ContainerStarted","Data":"ed50e7024e314c7c9b2738185e487045374a6c3a6bcb85e18690ecd5fbf9fd49"} Oct 30 11:32:17.167210 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:17.167132 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" Oct 30 11:32:17.192257 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:17.192225 4289 generic.go:356] "Generic (PLEG): container finished" podID="ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e" containerID="4fabd20cba075c0d4650b9abdaea12702060682fa315f9d440d76709d16afba9" exitCode=0 Oct 30 11:32:17.192485 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:17.192258 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e","Type":"ContainerDied","Data":"4fabd20cba075c0d4650b9abdaea12702060682fa315f9d440d76709d16afba9"} Oct 30 11:32:17.194086 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:17.194061 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"cfa1bffd-a96d-493b-89d8-76406e3e054d","Type":"ContainerStarted","Data":"ce57debd35bcf62d4ce902550664d14fcddf8706abb3cfb5d9b7d18f81f7ff41"} Oct 30 11:32:18.201067 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:18.200981 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e","Type":"ContainerStarted","Data":"685156e56f8b61506cf80a78eeb80a9e72787c001021e6160195ce16adc94300"} Oct 30 11:32:18.201067 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:18.201047 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e","Type":"ContainerStarted","Data":"6092255c5960182ac4874689d074fb9a3d9f442d5d54f719f1c7615b45ec3f37"} Oct 30 11:32:18.201523 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:18.201355 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:18.256443 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:18.256380 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=2.770264758 podStartE2EDuration="6.256364871s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.486100113s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:32:18.253502995 +0000 UTC m=+2828.754882480" watchObservedRunningTime="2026-10-30 11:32:18.256364871 +0000 UTC m=+2828.757744356" Oct 30 11:32:20.157299 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:20.157260 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r" Oct 30 11:32:20.210740 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:20.210708 4289 generic.go:356] "Generic (PLEG): container finished" podID="cfa1bffd-a96d-493b-89d8-76406e3e054d" containerID="ce57debd35bcf62d4ce902550664d14fcddf8706abb3cfb5d9b7d18f81f7ff41" exitCode=0 Oct 30 11:32:20.210895 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:20.210784 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"cfa1bffd-a96d-493b-89d8-76406e3e054d","Type":"ContainerDied","Data":"ce57debd35bcf62d4ce902550664d14fcddf8706abb3cfb5d9b7d18f81f7ff41"} Oct 30 11:32:21.222255 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:21.222215 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"cfa1bffd-a96d-493b-89d8-76406e3e054d","Type":"ContainerStarted","Data":"7f8d2731d6f986733b0185fc31a99ae6777b64184e4caf72eb680bcc41c99698"} Oct 30 11:32:21.222666 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:21.222305 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"cfa1bffd-a96d-493b-89d8-76406e3e054d","Type":"ContainerStarted","Data":"28025530848b3ec03431181086ab229472e16623f3e3e09b1e53652bdab041eb"} Oct 30 11:32:21.222666 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:21.222365 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"cfa1bffd-a96d-493b-89d8-76406e3e054d","Type":"ContainerStarted","Data":"32636af6b57cd66459c23494210ab983a7714eb9b6112ab98e3c1f12b7f27099"} Oct 30 11:32:21.302373 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:21.302313 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=5.517749407 podStartE2EDuration="9.302297937s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.78454853s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:32:21.300875852 +0000 UTC m=+2831.802255337" watchObservedRunningTime="2026-10-30 11:32:21.302297937 +0000 UTC m=+2831.803677421" Oct 30 11:32:24.206419 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:24.206337 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:32:25.539547 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:25.539510 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:26.256863 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:32:26.256827 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:32:30.540247 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:30.540211 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:30.545742 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:30.545717 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:31.271953 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:31.271924 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:40.795362 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:40.795324 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:41.101943 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:41.101843 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:41.301883 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:41.301843 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:42.046917 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.046880 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:32:42.047444 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.047254 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="cfa1bffd-a96d-493b-89d8-76406e3e054d" containerName="thanos-sidecar" containerID="cri-o://7f8d2731d6f986733b0185fc31a99ae6777b64184e4caf72eb680bcc41c99698" gracePeriod=600 Oct 30 11:32:42.047444 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.047200 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="cfa1bffd-a96d-493b-89d8-76406e3e054d" containerName="prometheus" containerID="cri-o://32636af6b57cd66459c23494210ab983a7714eb9b6112ab98e3c1f12b7f27099" gracePeriod=600 Oct 30 11:32:42.047444 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.047240 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="cfa1bffd-a96d-493b-89d8-76406e3e054d" containerName="config-reloader" containerID="cri-o://28025530848b3ec03431181086ab229472e16623f3e3e09b1e53652bdab041eb" gracePeriod=600 Oct 30 11:32:42.300758 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.300697 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:42.315828 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.315792 4289 generic.go:356] "Generic (PLEG): container finished" podID="cfa1bffd-a96d-493b-89d8-76406e3e054d" containerID="7f8d2731d6f986733b0185fc31a99ae6777b64184e4caf72eb680bcc41c99698" exitCode=0 Oct 30 11:32:42.315828 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.315819 4289 generic.go:356] "Generic (PLEG): container finished" podID="cfa1bffd-a96d-493b-89d8-76406e3e054d" containerID="28025530848b3ec03431181086ab229472e16623f3e3e09b1e53652bdab041eb" exitCode=0 Oct 30 11:32:42.315828 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.315828 4289 generic.go:356] "Generic (PLEG): container finished" podID="cfa1bffd-a96d-493b-89d8-76406e3e054d" containerID="32636af6b57cd66459c23494210ab983a7714eb9b6112ab98e3c1f12b7f27099" exitCode=0 Oct 30 11:32:42.316096 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.315879 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"cfa1bffd-a96d-493b-89d8-76406e3e054d","Type":"ContainerDied","Data":"7f8d2731d6f986733b0185fc31a99ae6777b64184e4caf72eb680bcc41c99698"} Oct 30 11:32:42.316096 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.315885 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:42.316096 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.315967 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"cfa1bffd-a96d-493b-89d8-76406e3e054d","Type":"ContainerDied","Data":"28025530848b3ec03431181086ab229472e16623f3e3e09b1e53652bdab041eb"} Oct 30 11:32:42.316096 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.315995 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"cfa1bffd-a96d-493b-89d8-76406e3e054d","Type":"ContainerDied","Data":"32636af6b57cd66459c23494210ab983a7714eb9b6112ab98e3c1f12b7f27099"} Oct 30 11:32:42.316096 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.316020 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"cfa1bffd-a96d-493b-89d8-76406e3e054d","Type":"ContainerDied","Data":"ed50e7024e314c7c9b2738185e487045374a6c3a6bcb85e18690ecd5fbf9fd49"} Oct 30 11:32:42.316096 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.316052 4289 scope.go:122] "RemoveContainer" containerID="7f8d2731d6f986733b0185fc31a99ae6777b64184e4caf72eb680bcc41c99698" Oct 30 11:32:42.324141 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.324125 4289 scope.go:122] "RemoveContainer" containerID="28025530848b3ec03431181086ab229472e16623f3e3e09b1e53652bdab041eb" Oct 30 11:32:42.331797 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.331781 4289 scope.go:122] "RemoveContainer" containerID="32636af6b57cd66459c23494210ab983a7714eb9b6112ab98e3c1f12b7f27099" Oct 30 11:32:42.339470 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.339449 4289 scope.go:122] "RemoveContainer" containerID="ce57debd35bcf62d4ce902550664d14fcddf8706abb3cfb5d9b7d18f81f7ff41" Oct 30 11:32:42.346799 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.346783 4289 scope.go:122] "RemoveContainer" containerID="7f8d2731d6f986733b0185fc31a99ae6777b64184e4caf72eb680bcc41c99698" Oct 30 11:32:42.347094 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:32:42.347069 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f8d2731d6f986733b0185fc31a99ae6777b64184e4caf72eb680bcc41c99698\": container with ID starting with 7f8d2731d6f986733b0185fc31a99ae6777b64184e4caf72eb680bcc41c99698 not found: ID does not exist" containerID="7f8d2731d6f986733b0185fc31a99ae6777b64184e4caf72eb680bcc41c99698" Oct 30 11:32:42.347146 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.347105 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f8d2731d6f986733b0185fc31a99ae6777b64184e4caf72eb680bcc41c99698"} err="failed to get container status \"7f8d2731d6f986733b0185fc31a99ae6777b64184e4caf72eb680bcc41c99698\": rpc error: code = NotFound desc = could not find container \"7f8d2731d6f986733b0185fc31a99ae6777b64184e4caf72eb680bcc41c99698\": container with ID starting with 7f8d2731d6f986733b0185fc31a99ae6777b64184e4caf72eb680bcc41c99698 not found: ID does not exist" Oct 30 11:32:42.347146 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.347123 4289 scope.go:122] "RemoveContainer" containerID="28025530848b3ec03431181086ab229472e16623f3e3e09b1e53652bdab041eb" Oct 30 11:32:42.347362 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:32:42.347339 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28025530848b3ec03431181086ab229472e16623f3e3e09b1e53652bdab041eb\": container with ID starting with 28025530848b3ec03431181086ab229472e16623f3e3e09b1e53652bdab041eb not found: ID does not exist" containerID="28025530848b3ec03431181086ab229472e16623f3e3e09b1e53652bdab041eb" Oct 30 11:32:42.347446 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.347373 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28025530848b3ec03431181086ab229472e16623f3e3e09b1e53652bdab041eb"} err="failed to get container status \"28025530848b3ec03431181086ab229472e16623f3e3e09b1e53652bdab041eb\": rpc error: code = NotFound desc = could not find container \"28025530848b3ec03431181086ab229472e16623f3e3e09b1e53652bdab041eb\": container with ID starting with 28025530848b3ec03431181086ab229472e16623f3e3e09b1e53652bdab041eb not found: ID does not exist" Oct 30 11:32:42.347446 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.347396 4289 scope.go:122] "RemoveContainer" containerID="32636af6b57cd66459c23494210ab983a7714eb9b6112ab98e3c1f12b7f27099" Oct 30 11:32:42.347640 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:32:42.347623 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32636af6b57cd66459c23494210ab983a7714eb9b6112ab98e3c1f12b7f27099\": container with ID starting with 32636af6b57cd66459c23494210ab983a7714eb9b6112ab98e3c1f12b7f27099 not found: ID does not exist" containerID="32636af6b57cd66459c23494210ab983a7714eb9b6112ab98e3c1f12b7f27099" Oct 30 11:32:42.347679 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.347645 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32636af6b57cd66459c23494210ab983a7714eb9b6112ab98e3c1f12b7f27099"} err="failed to get container status \"32636af6b57cd66459c23494210ab983a7714eb9b6112ab98e3c1f12b7f27099\": rpc error: code = NotFound desc = could not find container \"32636af6b57cd66459c23494210ab983a7714eb9b6112ab98e3c1f12b7f27099\": container with ID starting with 32636af6b57cd66459c23494210ab983a7714eb9b6112ab98e3c1f12b7f27099 not found: ID does not exist" Oct 30 11:32:42.347679 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.347661 4289 scope.go:122] "RemoveContainer" containerID="ce57debd35bcf62d4ce902550664d14fcddf8706abb3cfb5d9b7d18f81f7ff41" Oct 30 11:32:42.347912 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:32:42.347895 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce57debd35bcf62d4ce902550664d14fcddf8706abb3cfb5d9b7d18f81f7ff41\": container with ID starting with ce57debd35bcf62d4ce902550664d14fcddf8706abb3cfb5d9b7d18f81f7ff41 not found: ID does not exist" containerID="ce57debd35bcf62d4ce902550664d14fcddf8706abb3cfb5d9b7d18f81f7ff41" Oct 30 11:32:42.347980 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.347919 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce57debd35bcf62d4ce902550664d14fcddf8706abb3cfb5d9b7d18f81f7ff41"} err="failed to get container status \"ce57debd35bcf62d4ce902550664d14fcddf8706abb3cfb5d9b7d18f81f7ff41\": rpc error: code = NotFound desc = could not find container \"ce57debd35bcf62d4ce902550664d14fcddf8706abb3cfb5d9b7d18f81f7ff41\": container with ID starting with ce57debd35bcf62d4ce902550664d14fcddf8706abb3cfb5d9b7d18f81f7ff41 not found: ID does not exist" Oct 30 11:32:42.347980 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.347937 4289 scope.go:122] "RemoveContainer" containerID="7f8d2731d6f986733b0185fc31a99ae6777b64184e4caf72eb680bcc41c99698" Oct 30 11:32:42.348146 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.348130 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f8d2731d6f986733b0185fc31a99ae6777b64184e4caf72eb680bcc41c99698"} err="failed to get container status \"7f8d2731d6f986733b0185fc31a99ae6777b64184e4caf72eb680bcc41c99698\": rpc error: code = NotFound desc = could not find container \"7f8d2731d6f986733b0185fc31a99ae6777b64184e4caf72eb680bcc41c99698\": container with ID starting with 7f8d2731d6f986733b0185fc31a99ae6777b64184e4caf72eb680bcc41c99698 not found: ID does not exist" Oct 30 11:32:42.348192 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.348146 4289 scope.go:122] "RemoveContainer" containerID="28025530848b3ec03431181086ab229472e16623f3e3e09b1e53652bdab041eb" Oct 30 11:32:42.348338 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.348320 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28025530848b3ec03431181086ab229472e16623f3e3e09b1e53652bdab041eb"} err="failed to get container status \"28025530848b3ec03431181086ab229472e16623f3e3e09b1e53652bdab041eb\": rpc error: code = NotFound desc = could not find container \"28025530848b3ec03431181086ab229472e16623f3e3e09b1e53652bdab041eb\": container with ID starting with 28025530848b3ec03431181086ab229472e16623f3e3e09b1e53652bdab041eb not found: ID does not exist" Oct 30 11:32:42.348410 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.348339 4289 scope.go:122] "RemoveContainer" containerID="32636af6b57cd66459c23494210ab983a7714eb9b6112ab98e3c1f12b7f27099" Oct 30 11:32:42.348584 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.348565 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32636af6b57cd66459c23494210ab983a7714eb9b6112ab98e3c1f12b7f27099"} err="failed to get container status \"32636af6b57cd66459c23494210ab983a7714eb9b6112ab98e3c1f12b7f27099\": rpc error: code = NotFound desc = could not find container \"32636af6b57cd66459c23494210ab983a7714eb9b6112ab98e3c1f12b7f27099\": container with ID starting with 32636af6b57cd66459c23494210ab983a7714eb9b6112ab98e3c1f12b7f27099 not found: ID does not exist" Oct 30 11:32:42.348639 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.348586 4289 scope.go:122] "RemoveContainer" containerID="ce57debd35bcf62d4ce902550664d14fcddf8706abb3cfb5d9b7d18f81f7ff41" Oct 30 11:32:42.348780 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.348762 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce57debd35bcf62d4ce902550664d14fcddf8706abb3cfb5d9b7d18f81f7ff41"} err="failed to get container status \"ce57debd35bcf62d4ce902550664d14fcddf8706abb3cfb5d9b7d18f81f7ff41\": rpc error: code = NotFound desc = could not find container \"ce57debd35bcf62d4ce902550664d14fcddf8706abb3cfb5d9b7d18f81f7ff41\": container with ID starting with ce57debd35bcf62d4ce902550664d14fcddf8706abb3cfb5d9b7d18f81f7ff41 not found: ID does not exist" Oct 30 11:32:42.348849 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.348781 4289 scope.go:122] "RemoveContainer" containerID="7f8d2731d6f986733b0185fc31a99ae6777b64184e4caf72eb680bcc41c99698" Oct 30 11:32:42.348992 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.348976 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f8d2731d6f986733b0185fc31a99ae6777b64184e4caf72eb680bcc41c99698"} err="failed to get container status \"7f8d2731d6f986733b0185fc31a99ae6777b64184e4caf72eb680bcc41c99698\": rpc error: code = NotFound desc = could not find container \"7f8d2731d6f986733b0185fc31a99ae6777b64184e4caf72eb680bcc41c99698\": container with ID starting with 7f8d2731d6f986733b0185fc31a99ae6777b64184e4caf72eb680bcc41c99698 not found: ID does not exist" Oct 30 11:32:42.349037 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.348992 4289 scope.go:122] "RemoveContainer" containerID="28025530848b3ec03431181086ab229472e16623f3e3e09b1e53652bdab041eb" Oct 30 11:32:42.349171 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.349154 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28025530848b3ec03431181086ab229472e16623f3e3e09b1e53652bdab041eb"} err="failed to get container status \"28025530848b3ec03431181086ab229472e16623f3e3e09b1e53652bdab041eb\": rpc error: code = NotFound desc = could not find container \"28025530848b3ec03431181086ab229472e16623f3e3e09b1e53652bdab041eb\": container with ID starting with 28025530848b3ec03431181086ab229472e16623f3e3e09b1e53652bdab041eb not found: ID does not exist" Oct 30 11:32:42.349236 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.349171 4289 scope.go:122] "RemoveContainer" containerID="32636af6b57cd66459c23494210ab983a7714eb9b6112ab98e3c1f12b7f27099" Oct 30 11:32:42.349406 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.349389 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32636af6b57cd66459c23494210ab983a7714eb9b6112ab98e3c1f12b7f27099"} err="failed to get container status \"32636af6b57cd66459c23494210ab983a7714eb9b6112ab98e3c1f12b7f27099\": rpc error: code = NotFound desc = could not find container \"32636af6b57cd66459c23494210ab983a7714eb9b6112ab98e3c1f12b7f27099\": container with ID starting with 32636af6b57cd66459c23494210ab983a7714eb9b6112ab98e3c1f12b7f27099 not found: ID does not exist" Oct 30 11:32:42.349465 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.349407 4289 scope.go:122] "RemoveContainer" containerID="ce57debd35bcf62d4ce902550664d14fcddf8706abb3cfb5d9b7d18f81f7ff41" Oct 30 11:32:42.349656 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.349639 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce57debd35bcf62d4ce902550664d14fcddf8706abb3cfb5d9b7d18f81f7ff41"} err="failed to get container status \"ce57debd35bcf62d4ce902550664d14fcddf8706abb3cfb5d9b7d18f81f7ff41\": rpc error: code = NotFound desc = could not find container \"ce57debd35bcf62d4ce902550664d14fcddf8706abb3cfb5d9b7d18f81f7ff41\": container with ID starting with ce57debd35bcf62d4ce902550664d14fcddf8706abb3cfb5d9b7d18f81f7ff41 not found: ID does not exist" Oct 30 11:32:42.373331 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.373306 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-thanos-prometheus-http-client-file\") pod \"cfa1bffd-a96d-493b-89d8-76406e3e054d\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " Oct 30 11:32:42.373569 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.373353 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"cfa1bffd-a96d-493b-89d8-76406e3e054d\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " Oct 30 11:32:42.373569 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.373471 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"cfa1bffd-a96d-493b-89d8-76406e3e054d\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " Oct 30 11:32:42.373569 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.373523 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-web-config\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-web-config\") pod \"cfa1bffd-a96d-493b-89d8-76406e3e054d\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " Oct 30 11:32:42.373569 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.373557 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfa1bffd-a96d-493b-89d8-76406e3e054d-tls-assets\" (UniqueName: \"kubernetes.io/projected/cfa1bffd-a96d-493b-89d8-76406e3e054d-tls-assets\") pod \"cfa1bffd-a96d-493b-89d8-76406e3e054d\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " Oct 30 11:32:42.373812 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.373601 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfa1bffd-a96d-493b-89d8-76406e3e054d-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cfa1bffd-a96d-493b-89d8-76406e3e054d-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"cfa1bffd-a96d-493b-89d8-76406e3e054d\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " Oct 30 11:32:42.373812 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.373635 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfa1bffd-a96d-493b-89d8-76406e3e054d-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cfa1bffd-a96d-493b-89d8-76406e3e054d-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"cfa1bffd-a96d-493b-89d8-76406e3e054d\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " Oct 30 11:32:42.373812 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.373709 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"cfa1bffd-a96d-493b-89d8-76406e3e054d\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " Oct 30 11:32:42.373812 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.373756 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfa1bffd-a96d-493b-89d8-76406e3e054d-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cfa1bffd-a96d-493b-89d8-76406e3e054d-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"cfa1bffd-a96d-493b-89d8-76406e3e054d\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " Oct 30 11:32:42.373812 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.373788 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfa1bffd-a96d-493b-89d8-76406e3e054d-kube-api-access-f9tnh\" (UniqueName: \"kubernetes.io/projected/cfa1bffd-a96d-493b-89d8-76406e3e054d-kube-api-access-f9tnh\") pod \"cfa1bffd-a96d-493b-89d8-76406e3e054d\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " Oct 30 11:32:42.374087 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.373821 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfa1bffd-a96d-493b-89d8-76406e3e054d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/cfa1bffd-a96d-493b-89d8-76406e3e054d-config-out\") pod \"cfa1bffd-a96d-493b-89d8-76406e3e054d\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " Oct 30 11:32:42.374087 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.373870 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-secret-prometheus-web-tls-ca\") pod \"cfa1bffd-a96d-493b-89d8-76406e3e054d\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " Oct 30 11:32:42.374087 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.373903 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-config\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-config\") pod \"cfa1bffd-a96d-493b-89d8-76406e3e054d\" (UID: \"cfa1bffd-a96d-493b-89d8-76406e3e054d\") " Oct 30 11:32:42.374087 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.374050 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa1bffd-a96d-493b-89d8-76406e3e054d-prometheus-data-science-monitoringstack-rulefiles-1" pod "cfa1bffd-a96d-493b-89d8-76406e3e054d" (UID: "cfa1bffd-a96d-493b-89d8-76406e3e054d"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:32:42.375048 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.374785 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa1bffd-a96d-493b-89d8-76406e3e054d-prometheus-data-science-monitoringstack-rulefiles-2" pod "cfa1bffd-a96d-493b-89d8-76406e3e054d" (UID: "cfa1bffd-a96d-493b-89d8-76406e3e054d"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:32:42.375748 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.375383 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa1bffd-a96d-493b-89d8-76406e3e054d-prometheus-data-science-monitoringstack-rulefiles-0" pod "cfa1bffd-a96d-493b-89d8-76406e3e054d" (UID: "cfa1bffd-a96d-493b-89d8-76406e3e054d"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:32:42.376407 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.376387 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cfa1bffd-a96d-493b-89d8-76406e3e054d-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:32:42.376584 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.376566 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cfa1bffd-a96d-493b-89d8-76406e3e054d-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:32:42.376716 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.376701 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cfa1bffd-a96d-493b-89d8-76406e3e054d-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:32:42.376954 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.376887 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "cfa1bffd-a96d-493b-89d8-76406e3e054d" (UID: "cfa1bffd-a96d-493b-89d8-76406e3e054d"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:32:42.377466 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.377406 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-thanos-prometheus-http-client-file" pod "cfa1bffd-a96d-493b-89d8-76406e3e054d" (UID: "cfa1bffd-a96d-493b-89d8-76406e3e054d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:32:42.378039 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.377989 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa1bffd-a96d-493b-89d8-76406e3e054d-config-out" pod "cfa1bffd-a96d-493b-89d8-76406e3e054d" (UID: "cfa1bffd-a96d-493b-89d8-76406e3e054d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:32:42.378343 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.378318 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-config" pod "cfa1bffd-a96d-493b-89d8-76406e3e054d" (UID: "cfa1bffd-a96d-493b-89d8-76406e3e054d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:32:42.379349 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.379316 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "cfa1bffd-a96d-493b-89d8-76406e3e054d" (UID: "cfa1bffd-a96d-493b-89d8-76406e3e054d"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:32:42.379720 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.379674 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa1bffd-a96d-493b-89d8-76406e3e054d-tls-assets" pod "cfa1bffd-a96d-493b-89d8-76406e3e054d" (UID: "cfa1bffd-a96d-493b-89d8-76406e3e054d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:32:42.380605 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.380580 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48" pod "cfa1bffd-a96d-493b-89d8-76406e3e054d" (UID: "cfa1bffd-a96d-493b-89d8-76406e3e054d"). InnerVolumeSpecName "pvc-c799cc48-125a-421c-8fc6-8a4d017a7653". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 11:32:42.380942 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.380918 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa1bffd-a96d-493b-89d8-76406e3e054d-kube-api-access-f9tnh" pod "cfa1bffd-a96d-493b-89d8-76406e3e054d" (UID: "cfa1bffd-a96d-493b-89d8-76406e3e054d"). InnerVolumeSpecName "kube-api-access-f9tnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:32:42.381055 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.381011 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-secret-prometheus-web-tls-ca" pod "cfa1bffd-a96d-493b-89d8-76406e3e054d" (UID: "cfa1bffd-a96d-493b-89d8-76406e3e054d"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:32:42.390393 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.390358 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-web-config" pod "cfa1bffd-a96d-493b-89d8-76406e3e054d" (UID: "cfa1bffd-a96d-493b-89d8-76406e3e054d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:32:42.478136 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.478102 4289 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:32:42.478136 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.478132 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9tnh\" (UniqueName: \"kubernetes.io/projected/cfa1bffd-a96d-493b-89d8-76406e3e054d-kube-api-access-f9tnh\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:32:42.478136 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.478142 4289 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cfa1bffd-a96d-493b-89d8-76406e3e054d-config-out\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:32:42.478466 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.478151 4289 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-secret-prometheus-web-tls-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:32:42.478466 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.478160 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:32:42.478466 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.478170 4289 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:32:42.478466 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.478179 4289 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:32:42.478466 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.478207 4289 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") on node \"ip-10-0-137-242.ec2.internal\" " Oct 30 11:32:42.478466 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.478218 4289 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cfa1bffd-a96d-493b-89d8-76406e3e054d-web-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:32:42.478466 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.478225 4289 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cfa1bffd-a96d-493b-89d8-76406e3e054d-tls-assets\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:32:42.489690 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.489660 4289 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-c799cc48-125a-421c-8fc6-8a4d017a7653" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48") on node "ip-10-0-137-242.ec2.internal" Oct 30 11:32:42.579074 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.578991 4289 reconciler_common.go:299] "Volume detached for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"csi-ea06c66241a0a003a5daff5d9fc602f40e53768b00b104c5e4118b92d558c090\"" Oct 30 11:32:42.668755 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.668720 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:32:42.674804 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.674781 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:32:42.681508 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.681478 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:42.899273 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:42.899169 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:43.624313 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:43.624283 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfa1bffd-a96d-493b-89d8-76406e3e054d" path="/var/lib/kubelet/pods/cfa1bffd-a96d-493b-89d8-76406e3e054d/volumes" Oct 30 11:32:45.176690 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.176657 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:32:45.177080 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.177025 4289 cpu_manager.go:414] "containerMap: removing container" podUID="cfa1bffd-a96d-493b-89d8-76406e3e054d" containerName="config-reloader" Oct 30 11:32:45.177080 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.177037 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa1bffd-a96d-493b-89d8-76406e3e054d" containerName="config-reloader" Oct 30 11:32:45.177080 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.177055 4289 cpu_manager.go:414] "containerMap: removing container" podUID="cfa1bffd-a96d-493b-89d8-76406e3e054d" containerName="thanos-sidecar" Oct 30 11:32:45.177080 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.177059 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa1bffd-a96d-493b-89d8-76406e3e054d" containerName="thanos-sidecar" Oct 30 11:32:45.177080 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.177068 4289 cpu_manager.go:414] "containerMap: removing container" podUID="cfa1bffd-a96d-493b-89d8-76406e3e054d" containerName="prometheus" Oct 30 11:32:45.177080 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.177073 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa1bffd-a96d-493b-89d8-76406e3e054d" containerName="prometheus" Oct 30 11:32:45.177080 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.177080 4289 cpu_manager.go:414] "containerMap: removing container" podUID="cfa1bffd-a96d-493b-89d8-76406e3e054d" containerName="init-config-reloader" Oct 30 11:32:45.177080 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.177085 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa1bffd-a96d-493b-89d8-76406e3e054d" containerName="init-config-reloader" Oct 30 11:32:45.177336 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.177144 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa1bffd-a96d-493b-89d8-76406e3e054d" containerName="config-reloader" Oct 30 11:32:45.177336 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.177154 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa1bffd-a96d-493b-89d8-76406e3e054d" containerName="thanos-sidecar" Oct 30 11:32:45.177336 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.177160 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa1bffd-a96d-493b-89d8-76406e3e054d" containerName="prometheus" Oct 30 11:32:45.181615 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.181600 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:32:45.211582 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.211549 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-e4cadd38\"" Oct 30 11:32:45.211882 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.211861 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-2cltz\"" Oct 30 11:32:45.211977 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.211933 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 11:32:45.213004 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.212981 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:32:45.223995 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.223967 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 11:32:45.224092 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.224003 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 11:32:45.300198 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.300167 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a69e1e25-95cf-4907-81af-69c0c7b545a4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a69e1e25-95cf-4907-81af-69c0c7b545a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:32:45.300336 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.300197 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a69e1e25-95cf-4907-81af-69c0c7b545a4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"a69e1e25-95cf-4907-81af-69c0c7b545a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:32:45.300336 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.300230 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jtmj\" (UniqueName: \"kubernetes.io/projected/a69e1e25-95cf-4907-81af-69c0c7b545a4-kube-api-access-6jtmj\") pod \"data-science-collector-collector-1\" (UID: \"a69e1e25-95cf-4907-81af-69c0c7b545a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:32:45.300452 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.300337 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a69e1e25-95cf-4907-81af-69c0c7b545a4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a69e1e25-95cf-4907-81af-69c0c7b545a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:32:45.300452 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.300381 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a69e1e25-95cf-4907-81af-69c0c7b545a4-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a69e1e25-95cf-4907-81af-69c0c7b545a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:32:45.401505 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.401475 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a69e1e25-95cf-4907-81af-69c0c7b545a4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a69e1e25-95cf-4907-81af-69c0c7b545a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:32:45.401675 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.401520 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a69e1e25-95cf-4907-81af-69c0c7b545a4-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a69e1e25-95cf-4907-81af-69c0c7b545a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:32:45.401675 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.401577 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a69e1e25-95cf-4907-81af-69c0c7b545a4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a69e1e25-95cf-4907-81af-69c0c7b545a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:32:45.401675 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.401606 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a69e1e25-95cf-4907-81af-69c0c7b545a4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"a69e1e25-95cf-4907-81af-69c0c7b545a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:32:45.401675 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.401628 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jtmj\" (UniqueName: \"kubernetes.io/projected/a69e1e25-95cf-4907-81af-69c0c7b545a4-kube-api-access-6jtmj\") pod \"data-science-collector-collector-1\" (UID: \"a69e1e25-95cf-4907-81af-69c0c7b545a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:32:45.402354 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.402332 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a69e1e25-95cf-4907-81af-69c0c7b545a4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a69e1e25-95cf-4907-81af-69c0c7b545a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:32:45.404169 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.404145 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a69e1e25-95cf-4907-81af-69c0c7b545a4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a69e1e25-95cf-4907-81af-69c0c7b545a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:32:45.404334 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.404229 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a69e1e25-95cf-4907-81af-69c0c7b545a4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"a69e1e25-95cf-4907-81af-69c0c7b545a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:32:45.404412 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.404394 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a69e1e25-95cf-4907-81af-69c0c7b545a4-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a69e1e25-95cf-4907-81af-69c0c7b545a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:32:45.425691 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.425668 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jtmj\" (UniqueName: \"kubernetes.io/projected/a69e1e25-95cf-4907-81af-69c0c7b545a4-kube-api-access-6jtmj\") pod \"data-science-collector-collector-1\" (UID: \"a69e1e25-95cf-4907-81af-69c0c7b545a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:32:45.493882 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.493851 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:32:45.792330 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.792211 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:32:45.795033 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:32:45.795000 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda69e1e25_95cf_4907_81af_69c0c7b545a4.slice/crio-2db6bbd135343d09697d83c088b6cffdbbea43f0985640e4d7978b18de1f3c43 WatchSource:0}: Error finding container 2db6bbd135343d09697d83c088b6cffdbbea43f0985640e4d7978b18de1f3c43: Status 404 returned error can't find the container with id 2db6bbd135343d09697d83c088b6cffdbbea43f0985640e4d7978b18de1f3c43 Oct 30 11:32:45.797255 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.797221 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:32:45.797315 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:45.797285 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:32:46.336561 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:46.336528 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a69e1e25-95cf-4907-81af-69c0c7b545a4","Type":"ContainerStarted","Data":"32913bcee16deb087112d1fa0748846e30068c64d3b5d4c95a401372440ddfbd"} Oct 30 11:32:46.336995 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:46.336576 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a69e1e25-95cf-4907-81af-69c0c7b545a4","Type":"ContainerStarted","Data":"2db6bbd135343d09697d83c088b6cffdbbea43f0985640e4d7978b18de1f3c43"} Oct 30 11:32:46.379626 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:46.379575 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.379561281 podStartE2EDuration="1.379561281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:32:46.378413497 +0000 UTC m=+2856.879792992" watchObservedRunningTime="2026-10-30 11:32:46.379561281 +0000 UTC m=+2856.880940766" Oct 30 11:32:48.033244 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:48.033202 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:48.257896 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:48.257854 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:48.455979 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:48.455881 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:49.344068 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.344034 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:32:49.348127 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.348092 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.364722 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.364681 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:32:49.375085 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.375044 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 11:32:49.375085 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.375080 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 11:32:49.375360 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.375243 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 11:32:49.375360 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.375243 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 11:32:49.375360 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.375265 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 11:32:49.375360 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.375273 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 11:32:49.375360 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.375288 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 11:32:49.375360 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.375305 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 11:32:49.375360 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.375316 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-2lb6p\"" Oct 30 11:32:49.438005 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.437972 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.438005 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.438005 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.438205 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.438026 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.438205 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.438085 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7drp4\" (UniqueName: \"kubernetes.io/projected/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-kube-api-access-7drp4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.438205 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.438112 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.438205 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.438146 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.438344 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.438216 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.438344 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.438273 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.438410 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.438353 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.438410 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.438376 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.438493 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.438419 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.438529 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.438501 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.539380 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.539344 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.539380 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.539381 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.539647 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.539400 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.539647 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.539420 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7drp4\" (UniqueName: \"kubernetes.io/projected/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-kube-api-access-7drp4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.539647 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.539468 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.539647 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.539497 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.539647 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.539516 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.539647 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.539568 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.539647 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.539599 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.539647 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.539624 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.540066 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.539666 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.540066 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.539691 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.540677 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.540299 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.540677 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.540302 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.540883 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.540686 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.542517 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.542448 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.542840 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.542812 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.542939 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.542867 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.543014 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.543000 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.543136 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.543108 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.543238 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.543175 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.543461 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.543418 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.543548 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.543525 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.562946 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.562921 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7drp4\" (UniqueName: \"kubernetes.io/projected/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-kube-api-access-7drp4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:49.681250 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.681167 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:32:49.681401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.681357 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="a69e1e25-95cf-4907-81af-69c0c7b545a4" containerName="otc-container" containerID="cri-o://32913bcee16deb087112d1fa0748846e30068c64d3b5d4c95a401372440ddfbd" gracePeriod=30 Oct 30 11:32:49.873100 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.873065 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:49.923397 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:49.923366 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:32:50.037907 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.037876 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:50.043769 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.043745 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a69e1e25-95cf-4907-81af-69c0c7b545a4-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a69e1e25-95cf-4907-81af-69c0c7b545a4-monitoring-tls-certs\") pod \"a69e1e25-95cf-4907-81af-69c0c7b545a4\" (UID: \"a69e1e25-95cf-4907-81af-69c0c7b545a4\") " Oct 30 11:32:50.043869 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.043786 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a69e1e25-95cf-4907-81af-69c0c7b545a4-tls-certs\" (UniqueName: \"kubernetes.io/secret/a69e1e25-95cf-4907-81af-69c0c7b545a4-tls-certs\") pod \"a69e1e25-95cf-4907-81af-69c0c7b545a4\" (UID: \"a69e1e25-95cf-4907-81af-69c0c7b545a4\") " Oct 30 11:32:50.043869 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.043812 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a69e1e25-95cf-4907-81af-69c0c7b545a4-otc-internal\" (UniqueName: \"kubernetes.io/configmap/a69e1e25-95cf-4907-81af-69c0c7b545a4-otc-internal\") pod \"a69e1e25-95cf-4907-81af-69c0c7b545a4\" (UID: \"a69e1e25-95cf-4907-81af-69c0c7b545a4\") " Oct 30 11:32:50.043948 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.043900 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a69e1e25-95cf-4907-81af-69c0c7b545a4-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a69e1e25-95cf-4907-81af-69c0c7b545a4-data-science-collector-ta-client-cert\") pod \"a69e1e25-95cf-4907-81af-69c0c7b545a4\" (UID: \"a69e1e25-95cf-4907-81af-69c0c7b545a4\") " Oct 30 11:32:50.043983 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.043951 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a69e1e25-95cf-4907-81af-69c0c7b545a4-kube-api-access-6jtmj\" (UniqueName: \"kubernetes.io/projected/a69e1e25-95cf-4907-81af-69c0c7b545a4-kube-api-access-6jtmj\") pod \"a69e1e25-95cf-4907-81af-69c0c7b545a4\" (UID: \"a69e1e25-95cf-4907-81af-69c0c7b545a4\") " Oct 30 11:32:50.044265 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.044234 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a69e1e25-95cf-4907-81af-69c0c7b545a4-otc-internal" pod "a69e1e25-95cf-4907-81af-69c0c7b545a4" (UID: "a69e1e25-95cf-4907-81af-69c0c7b545a4"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:32:50.045944 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.045915 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a69e1e25-95cf-4907-81af-69c0c7b545a4-monitoring-tls-certs" pod "a69e1e25-95cf-4907-81af-69c0c7b545a4" (UID: "a69e1e25-95cf-4907-81af-69c0c7b545a4"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:32:50.046053 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.045975 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a69e1e25-95cf-4907-81af-69c0c7b545a4-data-science-collector-ta-client-cert" pod "a69e1e25-95cf-4907-81af-69c0c7b545a4" (UID: "a69e1e25-95cf-4907-81af-69c0c7b545a4"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:32:50.046053 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.045991 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a69e1e25-95cf-4907-81af-69c0c7b545a4-tls-certs" pod "a69e1e25-95cf-4907-81af-69c0c7b545a4" (UID: "a69e1e25-95cf-4907-81af-69c0c7b545a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:32:50.046053 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.046005 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a69e1e25-95cf-4907-81af-69c0c7b545a4-kube-api-access-6jtmj" pod "a69e1e25-95cf-4907-81af-69c0c7b545a4" (UID: "a69e1e25-95cf-4907-81af-69c0c7b545a4"). InnerVolumeSpecName "kube-api-access-6jtmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:32:50.144919 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.144880 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a69e1e25-95cf-4907-81af-69c0c7b545a4-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:32:50.144919 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.144914 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jtmj\" (UniqueName: \"kubernetes.io/projected/a69e1e25-95cf-4907-81af-69c0c7b545a4-kube-api-access-6jtmj\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:32:50.145128 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.144928 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a69e1e25-95cf-4907-81af-69c0c7b545a4-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:32:50.145128 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.144939 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a69e1e25-95cf-4907-81af-69c0c7b545a4-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:32:50.145128 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.144951 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a69e1e25-95cf-4907-81af-69c0c7b545a4-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:32:50.356731 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.356646 4289 generic.go:356] "Generic (PLEG): container finished" podID="a69e1e25-95cf-4907-81af-69c0c7b545a4" containerID="32913bcee16deb087112d1fa0748846e30068c64d3b5d4c95a401372440ddfbd" exitCode=0 Oct 30 11:32:50.356731 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.356693 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:32:50.356731 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.356713 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a69e1e25-95cf-4907-81af-69c0c7b545a4","Type":"ContainerDied","Data":"32913bcee16deb087112d1fa0748846e30068c64d3b5d4c95a401372440ddfbd"} Oct 30 11:32:50.357281 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.356753 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a69e1e25-95cf-4907-81af-69c0c7b545a4","Type":"ContainerDied","Data":"2db6bbd135343d09697d83c088b6cffdbbea43f0985640e4d7978b18de1f3c43"} Oct 30 11:32:50.357281 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.356773 4289 scope.go:122] "RemoveContainer" containerID="32913bcee16deb087112d1fa0748846e30068c64d3b5d4c95a401372440ddfbd" Oct 30 11:32:50.365747 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.365728 4289 scope.go:122] "RemoveContainer" containerID="32913bcee16deb087112d1fa0748846e30068c64d3b5d4c95a401372440ddfbd" Oct 30 11:32:50.365997 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:32:50.365977 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32913bcee16deb087112d1fa0748846e30068c64d3b5d4c95a401372440ddfbd\": container with ID starting with 32913bcee16deb087112d1fa0748846e30068c64d3b5d4c95a401372440ddfbd not found: ID does not exist" containerID="32913bcee16deb087112d1fa0748846e30068c64d3b5d4c95a401372440ddfbd" Oct 30 11:32:50.366056 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.366003 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32913bcee16deb087112d1fa0748846e30068c64d3b5d4c95a401372440ddfbd"} err="failed to get container status \"32913bcee16deb087112d1fa0748846e30068c64d3b5d4c95a401372440ddfbd\": rpc error: code = NotFound desc = could not find container \"32913bcee16deb087112d1fa0748846e30068c64d3b5d4c95a401372440ddfbd\": container with ID starting with 32913bcee16deb087112d1fa0748846e30068c64d3b5d4c95a401372440ddfbd not found: ID does not exist" Oct 30 11:32:50.407827 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.407798 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:32:50.411769 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:50.411745 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:32:51.624902 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:51.624865 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a69e1e25-95cf-4907-81af-69c0c7b545a4" path="/var/lib/kubelet/pods/a69e1e25-95cf-4907-81af-69c0c7b545a4/volumes" Oct 30 11:32:54.486350 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:54.486308 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:54.515717 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:54.515685 4289 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:54.587651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:54.587614 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:54.587804 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:54.587755 4289 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:54.602101 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:54.602077 4289 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") DevicePath \"csi-ea06c66241a0a003a5daff5d9fc602f40e53768b00b104c5e4118b92d558c090\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:54.704728 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:54.704697 4289 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7342da0537aa528cc1545d2018ac3fd0382dfe4f670067e58cf8ba34bb891174/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:54.733312 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:54.733283 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:54.769066 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:54.768995 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:32:55.099942 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:55.099862 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:32:55.114245 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:32:55.114217 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d00ed1_5c02_43b7_a670_e5061c0cbbd5.slice/crio-ab568a5735d50a6c3a239f543a889d23d72224bcd2e22d8ebe35d6e94f72311f WatchSource:0}: Error finding container ab568a5735d50a6c3a239f543a889d23d72224bcd2e22d8ebe35d6e94f72311f: Status 404 returned error can't find the container with id ab568a5735d50a6c3a239f543a889d23d72224bcd2e22d8ebe35d6e94f72311f Oct 30 11:32:55.380789 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:55.380705 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"05d00ed1-5c02-43b7-a670-e5061c0cbbd5","Type":"ContainerStarted","Data":"ab568a5735d50a6c3a239f543a889d23d72224bcd2e22d8ebe35d6e94f72311f"} Oct 30 11:32:56.258350 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:32:56.258316 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:32:56.387276 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:56.387236 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"05d00ed1-5c02-43b7-a670-e5061c0cbbd5","Type":"ContainerStarted","Data":"d075553409a49f786fcbb732e70895b503bdf3f7f7342235515116df4bfadf99"} Oct 30 11:32:58.984742 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:58.984694 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:32:59.401319 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:59.401286 4289 generic.go:356] "Generic (PLEG): container finished" podID="05d00ed1-5c02-43b7-a670-e5061c0cbbd5" containerID="d075553409a49f786fcbb732e70895b503bdf3f7f7342235515116df4bfadf99" exitCode=0 Oct 30 11:32:59.401507 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:32:59.401354 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"05d00ed1-5c02-43b7-a670-e5061c0cbbd5","Type":"ContainerDied","Data":"d075553409a49f786fcbb732e70895b503bdf3f7f7342235515116df4bfadf99"} Oct 30 11:33:00.408929 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:00.408836 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"05d00ed1-5c02-43b7-a670-e5061c0cbbd5","Type":"ContainerStarted","Data":"a9618d3674faef0b3a8f8665665279803c9b25ebc50715fc7747f729ded56ba2"} Oct 30 11:33:00.408929 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:00.408913 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"05d00ed1-5c02-43b7-a670-e5061c0cbbd5","Type":"ContainerStarted","Data":"b8b96c8f6fe7eee19ed7992ac90b36d3ae34829e2f1e294c51c30d4605e5a008"} Oct 30 11:33:00.409492 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:00.408951 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"05d00ed1-5c02-43b7-a670-e5061c0cbbd5","Type":"ContainerStarted","Data":"9484eb181fe0ae4005363c0db096ab13038ac17784c43d9a0daa85c69fa831c2"} Oct 30 11:33:00.487343 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:00.487288 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=8.20271556 podStartE2EDuration="11.487274841s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.284559281s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:33:00.486319049 +0000 UTC m=+2870.987698545" watchObservedRunningTime="2026-10-30 11:33:00.487274841 +0000 UTC m=+2870.988654326" Oct 30 11:33:04.770151 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:04.770110 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:33:09.769885 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:09.769853 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:33:09.775303 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:09.775281 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:33:10.459420 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:10.459346 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:33:13.261194 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:13.261160 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:13.507355 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:13.507319 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:13.692248 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:13.692156 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:15.133253 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:15.133219 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:15.307194 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:15.307163 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:18.519769 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:18.519736 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl"] Oct 30 11:33:18.520177 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:18.519971 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" podUID="62897926-e2e9-4408-b407-b1bdeb4ee0e9" containerName="kube-rbac-proxy" containerID="cri-o://dc31cad60c6be0b68da3b6460c0d2ebc0dfff678700947175123b025d1d1a6d2" gracePeriod=30 Oct 30 11:33:18.548831 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:18.548804 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl"] Oct 30 11:33:18.549067 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:18.549045 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" podUID="953398bd-24a9-4221-88bd-601a0796cf64" containerName="prom-label-proxy" containerID="cri-o://78520c074aa268c8db0a53ee76495756244a44435d3c7800ed40c4de29f8baf7" gracePeriod=30 Oct 30 11:33:18.549171 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:18.549049 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" podUID="953398bd-24a9-4221-88bd-601a0796cf64" containerName="kube-rbac-proxy" containerID="cri-o://0cbabac544cdd981b0e7d1042b9f8cef758eb32e37c0b4475a62b406557596dd" gracePeriod=30 Oct 30 11:33:18.549171 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:18.549073 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" podUID="953398bd-24a9-4221-88bd-601a0796cf64" containerName="method-gate" containerID="cri-o://f1837b2cdcc2402f8b98c1b33ec1d0f624b2024942a1964e405d945480d13140" gracePeriod=30 Oct 30 11:33:18.571521 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:18.571493 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w"] Oct 30 11:33:18.571824 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:18.571796 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" podUID="e9599347-1676-48b3-bf20-5ba203487718" containerName="prom-label-proxy" containerID="cri-o://845c046f53f477aadd9728ceecd8603c1c4b9dc74ef60ccd2fd84b83a79731fa" gracePeriod=30 Oct 30 11:33:18.571933 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:18.571795 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" podUID="e9599347-1676-48b3-bf20-5ba203487718" containerName="kube-rbac-proxy" containerID="cri-o://09347ec76edbb0b2c9acf02642b8e413dd84ef9bc7cd47461b8ed747cf3499f4" gracePeriod=30 Oct 30 11:33:18.571933 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:18.571819 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" podUID="e9599347-1676-48b3-bf20-5ba203487718" containerName="method-gate" containerID="cri-o://7e321e52776566c38897e08cefdc4a28f01959033fed400e7fb51fa4d1c3b125" gracePeriod=30 Oct 30 11:33:18.596067 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:18.596041 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r"] Oct 30 11:33:18.596365 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:18.596294 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r" podUID="ade4e9f0-3694-4382-a2e9-5105763562b6" containerName="korrel8r" containerID="cri-o://474b33182b99ee42a1fd1b4fa4974661aa72552251b840386f3136ddb3dbba03" gracePeriod=30 Oct 30 11:33:18.844183 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:18.844153 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" Oct 30 11:33:18.942473 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:18.942447 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" Oct 30 11:33:18.945136 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:18.945116 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" Oct 30 11:33:18.948702 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:18.948681 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r" Oct 30 11:33:19.009694 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.009638 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:19.016791 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.016760 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/953398bd-24a9-4221-88bd-601a0796cf64-client-ca\" (UniqueName: \"kubernetes.io/configmap/953398bd-24a9-4221-88bd-601a0796cf64-client-ca\") pod \"953398bd-24a9-4221-88bd-601a0796cf64\" (UID: \"953398bd-24a9-4221-88bd-601a0796cf64\") " Oct 30 11:33:19.016946 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.016813 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/953398bd-24a9-4221-88bd-601a0796cf64-tls\" (UniqueName: \"kubernetes.io/secret/953398bd-24a9-4221-88bd-601a0796cf64-tls\") pod \"953398bd-24a9-4221-88bd-601a0796cf64\" (UID: \"953398bd-24a9-4221-88bd-601a0796cf64\") " Oct 30 11:33:19.016946 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.016847 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/953398bd-24a9-4221-88bd-601a0796cf64-config\" (UniqueName: \"kubernetes.io/configmap/953398bd-24a9-4221-88bd-601a0796cf64-config\") pod \"953398bd-24a9-4221-88bd-601a0796cf64\" (UID: \"953398bd-24a9-4221-88bd-601a0796cf64\") " Oct 30 11:33:19.016946 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.016891 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/953398bd-24a9-4221-88bd-601a0796cf64-kube-api-access-pz52z\" (UniqueName: \"kubernetes.io/projected/953398bd-24a9-4221-88bd-601a0796cf64-kube-api-access-pz52z\") pod \"953398bd-24a9-4221-88bd-601a0796cf64\" (UID: \"953398bd-24a9-4221-88bd-601a0796cf64\") " Oct 30 11:33:19.017136 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.016960 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/953398bd-24a9-4221-88bd-601a0796cf64-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/953398bd-24a9-4221-88bd-601a0796cf64-prometheus-tls-ca\") pod \"953398bd-24a9-4221-88bd-601a0796cf64\" (UID: \"953398bd-24a9-4221-88bd-601a0796cf64\") " Oct 30 11:33:19.017328 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.017298 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/953398bd-24a9-4221-88bd-601a0796cf64-config" pod "953398bd-24a9-4221-88bd-601a0796cf64" (UID: "953398bd-24a9-4221-88bd-601a0796cf64"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:33:19.017485 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.017300 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/953398bd-24a9-4221-88bd-601a0796cf64-prometheus-tls-ca" pod "953398bd-24a9-4221-88bd-601a0796cf64" (UID: "953398bd-24a9-4221-88bd-601a0796cf64"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:33:19.017485 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.017471 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/953398bd-24a9-4221-88bd-601a0796cf64-client-ca" pod "953398bd-24a9-4221-88bd-601a0796cf64" (UID: "953398bd-24a9-4221-88bd-601a0796cf64"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:33:19.019180 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.019158 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/953398bd-24a9-4221-88bd-601a0796cf64-kube-api-access-pz52z" pod "953398bd-24a9-4221-88bd-601a0796cf64" (UID: "953398bd-24a9-4221-88bd-601a0796cf64"). InnerVolumeSpecName "kube-api-access-pz52z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:33:19.019277 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.019185 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/953398bd-24a9-4221-88bd-601a0796cf64-tls" pod "953398bd-24a9-4221-88bd-601a0796cf64" (UID: "953398bd-24a9-4221-88bd-601a0796cf64"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:33:19.058062 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.058024 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:33:19.058369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.058331 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e" containerName="alertmanager" containerID="cri-o://6092255c5960182ac4874689d074fb9a3d9f442d5d54f719f1c7615b45ec3f37" gracePeriod=120 Oct 30 11:33:19.058546 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.058399 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e" containerName="config-reloader" containerID="cri-o://685156e56f8b61506cf80a78eeb80a9e72787c001021e6160195ce16adc94300" gracePeriod=120 Oct 30 11:33:19.066465 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.066412 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:33:19.066797 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.066765 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="05d00ed1-5c02-43b7-a670-e5061c0cbbd5" containerName="prometheus" containerID="cri-o://9484eb181fe0ae4005363c0db096ab13038ac17784c43d9a0daa85c69fa831c2" gracePeriod=600 Oct 30 11:33:19.066990 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.066965 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="05d00ed1-5c02-43b7-a670-e5061c0cbbd5" containerName="thanos-sidecar" containerID="cri-o://a9618d3674faef0b3a8f8665665279803c9b25ebc50715fc7747f729ded56ba2" gracePeriod=600 Oct 30 11:33:19.067089 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.067069 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="05d00ed1-5c02-43b7-a670-e5061c0cbbd5" containerName="config-reloader" containerID="cri-o://b8b96c8f6fe7eee19ed7992ac90b36d3ae34829e2f1e294c51c30d4605e5a008" gracePeriod=600 Oct 30 11:33:19.118550 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.118514 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ade4e9f0-3694-4382-a2e9-5105763562b6-config\" (UniqueName: \"kubernetes.io/configmap/ade4e9f0-3694-4382-a2e9-5105763562b6-config\") pod \"ade4e9f0-3694-4382-a2e9-5105763562b6\" (UID: \"ade4e9f0-3694-4382-a2e9-5105763562b6\") " Oct 30 11:33:19.118746 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.118567 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9599347-1676-48b3-bf20-5ba203487718-kube-api-access-v97rg\" (UniqueName: \"kubernetes.io/projected/e9599347-1676-48b3-bf20-5ba203487718-kube-api-access-v97rg\") pod \"e9599347-1676-48b3-bf20-5ba203487718\" (UID: \"e9599347-1676-48b3-bf20-5ba203487718\") " Oct 30 11:33:19.118746 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.118613 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9599347-1676-48b3-bf20-5ba203487718-config\" (UniqueName: \"kubernetes.io/configmap/e9599347-1676-48b3-bf20-5ba203487718-config\") pod \"e9599347-1676-48b3-bf20-5ba203487718\" (UID: \"e9599347-1676-48b3-bf20-5ba203487718\") " Oct 30 11:33:19.118746 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.118637 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-tls\" (UniqueName: \"kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-tls\") pod \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\" (UID: \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\") " Oct 30 11:33:19.118746 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.118660 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62897926-e2e9-4408-b407-b1bdeb4ee0e9-kube-api-access-s2qlc\" (UniqueName: \"kubernetes.io/projected/62897926-e2e9-4408-b407-b1bdeb4ee0e9-kube-api-access-s2qlc\") pod \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\" (UID: \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\") " Oct 30 11:33:19.118746 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.118682 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ade4e9f0-3694-4382-a2e9-5105763562b6-tls\" (UniqueName: \"kubernetes.io/secret/ade4e9f0-3694-4382-a2e9-5105763562b6-tls\") pod \"ade4e9f0-3694-4382-a2e9-5105763562b6\" (UID: \"ade4e9f0-3694-4382-a2e9-5105763562b6\") " Oct 30 11:33:19.118746 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.118721 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-kube-rbac-proxy-config\") pod \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\" (UID: \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\") " Oct 30 11:33:19.119077 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.118758 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62897926-e2e9-4408-b407-b1bdeb4ee0e9-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/62897926-e2e9-4408-b407-b1bdeb4ee0e9-prometheus-ca\") pod \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\" (UID: \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\") " Oct 30 11:33:19.119077 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.118803 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade4e9f0-3694-4382-a2e9-5105763562b6-kube-api-access-mcz7g\" (UniqueName: \"kubernetes.io/projected/ade4e9f0-3694-4382-a2e9-5105763562b6-kube-api-access-mcz7g\") pod \"ade4e9f0-3694-4382-a2e9-5105763562b6\" (UID: \"ade4e9f0-3694-4382-a2e9-5105763562b6\") " Oct 30 11:33:19.119077 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.118837 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9599347-1676-48b3-bf20-5ba203487718-tls\" (UniqueName: \"kubernetes.io/secret/e9599347-1676-48b3-bf20-5ba203487718-tls\") pod \"e9599347-1676-48b3-bf20-5ba203487718\" (UID: \"e9599347-1676-48b3-bf20-5ba203487718\") " Oct 30 11:33:19.119077 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.118882 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-prometheus-client-cert\") pod \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\" (UID: \"62897926-e2e9-4408-b407-b1bdeb4ee0e9\") " Oct 30 11:33:19.119077 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.118923 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade4e9f0-3694-4382-a2e9-5105763562b6-config" pod "ade4e9f0-3694-4382-a2e9-5105763562b6" (UID: "ade4e9f0-3694-4382-a2e9-5105763562b6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:33:19.119336 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.119155 4289 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/953398bd-24a9-4221-88bd-601a0796cf64-client-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.119336 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.119177 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/953398bd-24a9-4221-88bd-601a0796cf64-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.119336 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.119190 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/953398bd-24a9-4221-88bd-601a0796cf64-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.119336 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.119203 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz52z\" (UniqueName: \"kubernetes.io/projected/953398bd-24a9-4221-88bd-601a0796cf64-kube-api-access-pz52z\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.119336 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.119220 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/953398bd-24a9-4221-88bd-601a0796cf64-prometheus-tls-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.119336 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.119234 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ade4e9f0-3694-4382-a2e9-5105763562b6-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.119336 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.119229 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9599347-1676-48b3-bf20-5ba203487718-config" pod "e9599347-1676-48b3-bf20-5ba203487718" (UID: "e9599347-1676-48b3-bf20-5ba203487718"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:33:19.119914 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.119885 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62897926-e2e9-4408-b407-b1bdeb4ee0e9-prometheus-ca" pod "62897926-e2e9-4408-b407-b1bdeb4ee0e9" (UID: "62897926-e2e9-4408-b407-b1bdeb4ee0e9"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:33:19.178293 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.177150 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-kube-rbac-proxy-config" pod "62897926-e2e9-4408-b407-b1bdeb4ee0e9" (UID: "62897926-e2e9-4408-b407-b1bdeb4ee0e9"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:33:19.178293 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.177542 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-tls" pod "62897926-e2e9-4408-b407-b1bdeb4ee0e9" (UID: "62897926-e2e9-4408-b407-b1bdeb4ee0e9"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:33:19.178293 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.177680 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ade4e9f0-3694-4382-a2e9-5105763562b6-tls" pod "ade4e9f0-3694-4382-a2e9-5105763562b6" (UID: "ade4e9f0-3694-4382-a2e9-5105763562b6"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:33:19.178293 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.177744 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62897926-e2e9-4408-b407-b1bdeb4ee0e9-kube-api-access-s2qlc" pod "62897926-e2e9-4408-b407-b1bdeb4ee0e9" (UID: "62897926-e2e9-4408-b407-b1bdeb4ee0e9"). InnerVolumeSpecName "kube-api-access-s2qlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:33:19.178293 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.177914 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-prometheus-client-cert" pod "62897926-e2e9-4408-b407-b1bdeb4ee0e9" (UID: "62897926-e2e9-4408-b407-b1bdeb4ee0e9"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:33:19.178293 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.177966 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9599347-1676-48b3-bf20-5ba203487718-tls" pod "e9599347-1676-48b3-bf20-5ba203487718" (UID: "e9599347-1676-48b3-bf20-5ba203487718"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:33:19.178843 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.178376 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade4e9f0-3694-4382-a2e9-5105763562b6-kube-api-access-mcz7g" pod "ade4e9f0-3694-4382-a2e9-5105763562b6" (UID: "ade4e9f0-3694-4382-a2e9-5105763562b6"). InnerVolumeSpecName "kube-api-access-mcz7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:33:19.178843 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.178462 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9599347-1676-48b3-bf20-5ba203487718-kube-api-access-v97rg" pod "e9599347-1676-48b3-bf20-5ba203487718" (UID: "e9599347-1676-48b3-bf20-5ba203487718"). InnerVolumeSpecName "kube-api-access-v97rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:33:19.190642 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.190493 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:19.206200 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.206167 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.107:9093/-/ready\": dial tcp 10.134.0.107:9093: connect: connection refused" Oct 30 11:33:19.220667 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.220642 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/62897926-e2e9-4408-b407-b1bdeb4ee0e9-prometheus-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.220667 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.220667 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcz7g\" (UniqueName: \"kubernetes.io/projected/ade4e9f0-3694-4382-a2e9-5105763562b6-kube-api-access-mcz7g\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.220824 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.220677 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e9599347-1676-48b3-bf20-5ba203487718-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.220824 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.220687 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-prometheus-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.220824 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.220697 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v97rg\" (UniqueName: \"kubernetes.io/projected/e9599347-1676-48b3-bf20-5ba203487718-kube-api-access-v97rg\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.220824 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.220705 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e9599347-1676-48b3-bf20-5ba203487718-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.220824 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.220713 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.220824 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.220721 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2qlc\" (UniqueName: \"kubernetes.io/projected/62897926-e2e9-4408-b407-b1bdeb4ee0e9-kube-api-access-s2qlc\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.220824 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.220730 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ade4e9f0-3694-4382-a2e9-5105763562b6-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.220824 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.220740 4289 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62897926-e2e9-4408-b407-b1bdeb4ee0e9-kube-rbac-proxy-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.357482 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.357449 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:19.408016 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.407992 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:33:19.502099 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.502065 4289 generic.go:356] "Generic (PLEG): container finished" podID="62897926-e2e9-4408-b407-b1bdeb4ee0e9" containerID="dc31cad60c6be0b68da3b6460c0d2ebc0dfff678700947175123b025d1d1a6d2" exitCode=0 Oct 30 11:33:19.502287 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.502117 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" Oct 30 11:33:19.502287 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.502145 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" event={"ID":"62897926-e2e9-4408-b407-b1bdeb4ee0e9","Type":"ContainerDied","Data":"dc31cad60c6be0b68da3b6460c0d2ebc0dfff678700947175123b025d1d1a6d2"} Oct 30 11:33:19.502287 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.502205 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl" event={"ID":"62897926-e2e9-4408-b407-b1bdeb4ee0e9","Type":"ContainerDied","Data":"efb8d4074b533e97225b3b2fe3f87cd9ccc7832ee8d3682e5a6452f5ac3f234f"} Oct 30 11:33:19.502287 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.502240 4289 scope.go:122] "RemoveContainer" containerID="dc31cad60c6be0b68da3b6460c0d2ebc0dfff678700947175123b025d1d1a6d2" Oct 30 11:33:19.504840 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.504814 4289 generic.go:356] "Generic (PLEG): container finished" podID="ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e" containerID="685156e56f8b61506cf80a78eeb80a9e72787c001021e6160195ce16adc94300" exitCode=0 Oct 30 11:33:19.504959 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.504891 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e","Type":"ContainerDied","Data":"685156e56f8b61506cf80a78eeb80a9e72787c001021e6160195ce16adc94300"} Oct 30 11:33:19.506298 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.506276 4289 generic.go:356] "Generic (PLEG): container finished" podID="ade4e9f0-3694-4382-a2e9-5105763562b6" containerID="474b33182b99ee42a1fd1b4fa4974661aa72552251b840386f3136ddb3dbba03" exitCode=0 Oct 30 11:33:19.506465 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.506320 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r" Oct 30 11:33:19.506465 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.506352 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r" event={"ID":"ade4e9f0-3694-4382-a2e9-5105763562b6","Type":"ContainerDied","Data":"474b33182b99ee42a1fd1b4fa4974661aa72552251b840386f3136ddb3dbba03"} Oct 30 11:33:19.506465 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.506403 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r" event={"ID":"ade4e9f0-3694-4382-a2e9-5105763562b6","Type":"ContainerDied","Data":"138684da53fc62550f29a0491425d9e6c94a392bf20701321c090601f9695c19"} Oct 30 11:33:19.511260 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.511236 4289 generic.go:356] "Generic (PLEG): container finished" podID="e9599347-1676-48b3-bf20-5ba203487718" containerID="845c046f53f477aadd9728ceecd8603c1c4b9dc74ef60ccd2fd84b83a79731fa" exitCode=0 Oct 30 11:33:19.511260 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.511259 4289 generic.go:356] "Generic (PLEG): container finished" podID="e9599347-1676-48b3-bf20-5ba203487718" containerID="7e321e52776566c38897e08cefdc4a28f01959033fed400e7fb51fa4d1c3b125" exitCode=0 Oct 30 11:33:19.511536 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.511279 4289 generic.go:356] "Generic (PLEG): container finished" podID="e9599347-1676-48b3-bf20-5ba203487718" containerID="09347ec76edbb0b2c9acf02642b8e413dd84ef9bc7cd47461b8ed747cf3499f4" exitCode=0 Oct 30 11:33:19.511536 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.511302 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" Oct 30 11:33:19.511536 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.511309 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" event={"ID":"e9599347-1676-48b3-bf20-5ba203487718","Type":"ContainerDied","Data":"845c046f53f477aadd9728ceecd8603c1c4b9dc74ef60ccd2fd84b83a79731fa"} Oct 30 11:33:19.511536 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.511369 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" event={"ID":"e9599347-1676-48b3-bf20-5ba203487718","Type":"ContainerDied","Data":"7e321e52776566c38897e08cefdc4a28f01959033fed400e7fb51fa4d1c3b125"} Oct 30 11:33:19.511536 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.511464 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" event={"ID":"e9599347-1676-48b3-bf20-5ba203487718","Type":"ContainerDied","Data":"09347ec76edbb0b2c9acf02642b8e413dd84ef9bc7cd47461b8ed747cf3499f4"} Oct 30 11:33:19.511536 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.511508 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w" event={"ID":"e9599347-1676-48b3-bf20-5ba203487718","Type":"ContainerDied","Data":"3aab1c3b3637a463594ff1bfcfbb3eef5180769ccff2cec008adc0b194cf7f2f"} Oct 30 11:33:19.515539 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.515519 4289 scope.go:122] "RemoveContainer" containerID="dc31cad60c6be0b68da3b6460c0d2ebc0dfff678700947175123b025d1d1a6d2" Oct 30 11:33:19.516090 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:19.516059 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc31cad60c6be0b68da3b6460c0d2ebc0dfff678700947175123b025d1d1a6d2\": container with ID starting with dc31cad60c6be0b68da3b6460c0d2ebc0dfff678700947175123b025d1d1a6d2 not found: ID does not exist" containerID="dc31cad60c6be0b68da3b6460c0d2ebc0dfff678700947175123b025d1d1a6d2" Oct 30 11:33:19.516194 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.516101 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc31cad60c6be0b68da3b6460c0d2ebc0dfff678700947175123b025d1d1a6d2"} err="failed to get container status \"dc31cad60c6be0b68da3b6460c0d2ebc0dfff678700947175123b025d1d1a6d2\": rpc error: code = NotFound desc = could not find container \"dc31cad60c6be0b68da3b6460c0d2ebc0dfff678700947175123b025d1d1a6d2\": container with ID starting with dc31cad60c6be0b68da3b6460c0d2ebc0dfff678700947175123b025d1d1a6d2 not found: ID does not exist" Oct 30 11:33:19.516194 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.516124 4289 scope.go:122] "RemoveContainer" containerID="474b33182b99ee42a1fd1b4fa4974661aa72552251b840386f3136ddb3dbba03" Oct 30 11:33:19.517958 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.517938 4289 generic.go:356] "Generic (PLEG): container finished" podID="953398bd-24a9-4221-88bd-601a0796cf64" containerID="78520c074aa268c8db0a53ee76495756244a44435d3c7800ed40c4de29f8baf7" exitCode=0 Oct 30 11:33:19.517958 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.517958 4289 generic.go:356] "Generic (PLEG): container finished" podID="953398bd-24a9-4221-88bd-601a0796cf64" containerID="f1837b2cdcc2402f8b98c1b33ec1d0f624b2024942a1964e405d945480d13140" exitCode=0 Oct 30 11:33:19.518113 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.517972 4289 generic.go:356] "Generic (PLEG): container finished" podID="953398bd-24a9-4221-88bd-601a0796cf64" containerID="0cbabac544cdd981b0e7d1042b9f8cef758eb32e37c0b4475a62b406557596dd" exitCode=0 Oct 30 11:33:19.518113 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.517994 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" Oct 30 11:33:19.518113 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.518021 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" event={"ID":"953398bd-24a9-4221-88bd-601a0796cf64","Type":"ContainerDied","Data":"78520c074aa268c8db0a53ee76495756244a44435d3c7800ed40c4de29f8baf7"} Oct 30 11:33:19.518113 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.518094 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" event={"ID":"953398bd-24a9-4221-88bd-601a0796cf64","Type":"ContainerDied","Data":"f1837b2cdcc2402f8b98c1b33ec1d0f624b2024942a1964e405d945480d13140"} Oct 30 11:33:19.518326 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.518141 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" event={"ID":"953398bd-24a9-4221-88bd-601a0796cf64","Type":"ContainerDied","Data":"0cbabac544cdd981b0e7d1042b9f8cef758eb32e37c0b4475a62b406557596dd"} Oct 30 11:33:19.518326 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.518185 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl" event={"ID":"953398bd-24a9-4221-88bd-601a0796cf64","Type":"ContainerDied","Data":"83bf9c8b4b995c12966f113f39c5ee125af7743a9d3ef8354f4f062c511eabb1"} Oct 30 11:33:19.521073 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.521015 4289 generic.go:356] "Generic (PLEG): container finished" podID="05d00ed1-5c02-43b7-a670-e5061c0cbbd5" containerID="a9618d3674faef0b3a8f8665665279803c9b25ebc50715fc7747f729ded56ba2" exitCode=0 Oct 30 11:33:19.521073 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.521035 4289 generic.go:356] "Generic (PLEG): container finished" podID="05d00ed1-5c02-43b7-a670-e5061c0cbbd5" containerID="b8b96c8f6fe7eee19ed7992ac90b36d3ae34829e2f1e294c51c30d4605e5a008" exitCode=0 Oct 30 11:33:19.521073 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.521044 4289 generic.go:356] "Generic (PLEG): container finished" podID="05d00ed1-5c02-43b7-a670-e5061c0cbbd5" containerID="9484eb181fe0ae4005363c0db096ab13038ac17784c43d9a0daa85c69fa831c2" exitCode=0 Oct 30 11:33:19.521545 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.521078 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"05d00ed1-5c02-43b7-a670-e5061c0cbbd5","Type":"ContainerDied","Data":"a9618d3674faef0b3a8f8665665279803c9b25ebc50715fc7747f729ded56ba2"} Oct 30 11:33:19.521545 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.521088 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:33:19.521545 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.521130 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"05d00ed1-5c02-43b7-a670-e5061c0cbbd5","Type":"ContainerDied","Data":"b8b96c8f6fe7eee19ed7992ac90b36d3ae34829e2f1e294c51c30d4605e5a008"} Oct 30 11:33:19.521545 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.521170 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"05d00ed1-5c02-43b7-a670-e5061c0cbbd5","Type":"ContainerDied","Data":"9484eb181fe0ae4005363c0db096ab13038ac17784c43d9a0daa85c69fa831c2"} Oct 30 11:33:19.521545 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.521216 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"05d00ed1-5c02-43b7-a670-e5061c0cbbd5","Type":"ContainerDied","Data":"ab568a5735d50a6c3a239f543a889d23d72224bcd2e22d8ebe35d6e94f72311f"} Oct 30 11:33:19.523294 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.523277 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-tls-assets\" (UniqueName: \"kubernetes.io/projected/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-tls-assets\") pod \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " Oct 30 11:33:19.523393 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.523326 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-config\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-config\") pod \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " Oct 30 11:33:19.523393 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.523359 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " Oct 30 11:33:19.523530 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.523418 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-thanos-prometheus-http-client-file\") pod \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " Oct 30 11:33:19.523589 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.523531 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " Oct 30 11:33:19.523650 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.523602 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-kube-api-access-7drp4\" (UniqueName: \"kubernetes.io/projected/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-kube-api-access-7drp4\") pod \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " Oct 30 11:33:19.523650 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.523637 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " Oct 30 11:33:19.523758 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.523681 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-config-out\" (UniqueName: \"kubernetes.io/empty-dir/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-config-out\") pod \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " Oct 30 11:33:19.523758 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.523709 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-secret-prometheus-web-tls-ca\") pod \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " Oct 30 11:33:19.523758 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.523748 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " Oct 30 11:33:19.523928 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.523781 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-web-config\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-web-config\") pod \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " Oct 30 11:33:19.523928 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.523824 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " Oct 30 11:33:19.523928 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.523864 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\" (UID: \"05d00ed1-5c02-43b7-a670-e5061c0cbbd5\") " Oct 30 11:33:19.524499 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.524478 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-prometheus-data-science-monitoringstack-rulefiles-0" pod "05d00ed1-5c02-43b7-a670-e5061c0cbbd5" (UID: "05d00ed1-5c02-43b7-a670-e5061c0cbbd5"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:33:19.525919 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.525892 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "05d00ed1-5c02-43b7-a670-e5061c0cbbd5" (UID: "05d00ed1-5c02-43b7-a670-e5061c0cbbd5"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:33:19.526009 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.525939 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-config" pod "05d00ed1-5c02-43b7-a670-e5061c0cbbd5" (UID: "05d00ed1-5c02-43b7-a670-e5061c0cbbd5"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:33:19.526196 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.526177 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-prometheus-data-science-monitoringstack-rulefiles-2" pod "05d00ed1-5c02-43b7-a670-e5061c0cbbd5" (UID: "05d00ed1-5c02-43b7-a670-e5061c0cbbd5"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:33:19.526273 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.526231 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-tls-assets" pod "05d00ed1-5c02-43b7-a670-e5061c0cbbd5" (UID: "05d00ed1-5c02-43b7-a670-e5061c0cbbd5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:33:19.526843 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.526594 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-prometheus-data-science-monitoringstack-rulefiles-1" pod "05d00ed1-5c02-43b7-a670-e5061c0cbbd5" (UID: "05d00ed1-5c02-43b7-a670-e5061c0cbbd5"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:33:19.527912 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.527893 4289 scope.go:122] "RemoveContainer" containerID="474b33182b99ee42a1fd1b4fa4974661aa72552251b840386f3136ddb3dbba03" Oct 30 11:33:19.528398 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:19.528377 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"474b33182b99ee42a1fd1b4fa4974661aa72552251b840386f3136ddb3dbba03\": container with ID starting with 474b33182b99ee42a1fd1b4fa4974661aa72552251b840386f3136ddb3dbba03 not found: ID does not exist" containerID="474b33182b99ee42a1fd1b4fa4974661aa72552251b840386f3136ddb3dbba03" Oct 30 11:33:19.528524 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.528406 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474b33182b99ee42a1fd1b4fa4974661aa72552251b840386f3136ddb3dbba03"} err="failed to get container status \"474b33182b99ee42a1fd1b4fa4974661aa72552251b840386f3136ddb3dbba03\": rpc error: code = NotFound desc = could not find container \"474b33182b99ee42a1fd1b4fa4974661aa72552251b840386f3136ddb3dbba03\": container with ID starting with 474b33182b99ee42a1fd1b4fa4974661aa72552251b840386f3136ddb3dbba03 not found: ID does not exist" Oct 30 11:33:19.528524 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.528446 4289 scope.go:122] "RemoveContainer" containerID="845c046f53f477aadd9728ceecd8603c1c4b9dc74ef60ccd2fd84b83a79731fa" Oct 30 11:33:19.528658 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.528518 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-kube-api-access-7drp4" pod "05d00ed1-5c02-43b7-a670-e5061c0cbbd5" (UID: "05d00ed1-5c02-43b7-a670-e5061c0cbbd5"). InnerVolumeSpecName "kube-api-access-7drp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:33:19.529713 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.529526 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-config-out" pod "05d00ed1-5c02-43b7-a670-e5061c0cbbd5" (UID: "05d00ed1-5c02-43b7-a670-e5061c0cbbd5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:33:19.529838 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.529812 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-thanos-prometheus-http-client-file" pod "05d00ed1-5c02-43b7-a670-e5061c0cbbd5" (UID: "05d00ed1-5c02-43b7-a670-e5061c0cbbd5"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:33:19.531339 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.531314 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-secret-prometheus-web-tls-ca" pod "05d00ed1-5c02-43b7-a670-e5061c0cbbd5" (UID: "05d00ed1-5c02-43b7-a670-e5061c0cbbd5"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:33:19.531480 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.531408 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "05d00ed1-5c02-43b7-a670-e5061c0cbbd5" (UID: "05d00ed1-5c02-43b7-a670-e5061c0cbbd5"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:33:19.532231 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.532213 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48" pod "05d00ed1-5c02-43b7-a670-e5061c0cbbd5" (UID: "05d00ed1-5c02-43b7-a670-e5061c0cbbd5"). InnerVolumeSpecName "pvc-c799cc48-125a-421c-8fc6-8a4d017a7653". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 11:33:19.537617 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.537599 4289 scope.go:122] "RemoveContainer" containerID="7e321e52776566c38897e08cefdc4a28f01959033fed400e7fb51fa4d1c3b125" Oct 30 11:33:19.540413 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.540392 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-web-config" pod "05d00ed1-5c02-43b7-a670-e5061c0cbbd5" (UID: "05d00ed1-5c02-43b7-a670-e5061c0cbbd5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:33:19.544888 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.544870 4289 scope.go:122] "RemoveContainer" containerID="09347ec76edbb0b2c9acf02642b8e413dd84ef9bc7cd47461b8ed747cf3499f4" Oct 30 11:33:19.552247 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.552225 4289 scope.go:122] "RemoveContainer" containerID="845c046f53f477aadd9728ceecd8603c1c4b9dc74ef60ccd2fd84b83a79731fa" Oct 30 11:33:19.552549 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:19.552531 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"845c046f53f477aadd9728ceecd8603c1c4b9dc74ef60ccd2fd84b83a79731fa\": container with ID starting with 845c046f53f477aadd9728ceecd8603c1c4b9dc74ef60ccd2fd84b83a79731fa not found: ID does not exist" containerID="845c046f53f477aadd9728ceecd8603c1c4b9dc74ef60ccd2fd84b83a79731fa" Oct 30 11:33:19.552602 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.552556 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"845c046f53f477aadd9728ceecd8603c1c4b9dc74ef60ccd2fd84b83a79731fa"} err="failed to get container status \"845c046f53f477aadd9728ceecd8603c1c4b9dc74ef60ccd2fd84b83a79731fa\": rpc error: code = NotFound desc = could not find container \"845c046f53f477aadd9728ceecd8603c1c4b9dc74ef60ccd2fd84b83a79731fa\": container with ID starting with 845c046f53f477aadd9728ceecd8603c1c4b9dc74ef60ccd2fd84b83a79731fa not found: ID does not exist" Oct 30 11:33:19.552602 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.552574 4289 scope.go:122] "RemoveContainer" containerID="7e321e52776566c38897e08cefdc4a28f01959033fed400e7fb51fa4d1c3b125" Oct 30 11:33:19.552827 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:19.552810 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e321e52776566c38897e08cefdc4a28f01959033fed400e7fb51fa4d1c3b125\": container with ID starting with 7e321e52776566c38897e08cefdc4a28f01959033fed400e7fb51fa4d1c3b125 not found: ID does not exist" containerID="7e321e52776566c38897e08cefdc4a28f01959033fed400e7fb51fa4d1c3b125" Oct 30 11:33:19.552879 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.552832 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e321e52776566c38897e08cefdc4a28f01959033fed400e7fb51fa4d1c3b125"} err="failed to get container status \"7e321e52776566c38897e08cefdc4a28f01959033fed400e7fb51fa4d1c3b125\": rpc error: code = NotFound desc = could not find container \"7e321e52776566c38897e08cefdc4a28f01959033fed400e7fb51fa4d1c3b125\": container with ID starting with 7e321e52776566c38897e08cefdc4a28f01959033fed400e7fb51fa4d1c3b125 not found: ID does not exist" Oct 30 11:33:19.552879 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.552844 4289 scope.go:122] "RemoveContainer" containerID="09347ec76edbb0b2c9acf02642b8e413dd84ef9bc7cd47461b8ed747cf3499f4" Oct 30 11:33:19.553053 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:19.553039 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09347ec76edbb0b2c9acf02642b8e413dd84ef9bc7cd47461b8ed747cf3499f4\": container with ID starting with 09347ec76edbb0b2c9acf02642b8e413dd84ef9bc7cd47461b8ed747cf3499f4 not found: ID does not exist" containerID="09347ec76edbb0b2c9acf02642b8e413dd84ef9bc7cd47461b8ed747cf3499f4" Oct 30 11:33:19.553088 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.553057 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09347ec76edbb0b2c9acf02642b8e413dd84ef9bc7cd47461b8ed747cf3499f4"} err="failed to get container status \"09347ec76edbb0b2c9acf02642b8e413dd84ef9bc7cd47461b8ed747cf3499f4\": rpc error: code = NotFound desc = could not find container \"09347ec76edbb0b2c9acf02642b8e413dd84ef9bc7cd47461b8ed747cf3499f4\": container with ID starting with 09347ec76edbb0b2c9acf02642b8e413dd84ef9bc7cd47461b8ed747cf3499f4 not found: ID does not exist" Oct 30 11:33:19.553088 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.553069 4289 scope.go:122] "RemoveContainer" containerID="845c046f53f477aadd9728ceecd8603c1c4b9dc74ef60ccd2fd84b83a79731fa" Oct 30 11:33:19.553303 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.553282 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"845c046f53f477aadd9728ceecd8603c1c4b9dc74ef60ccd2fd84b83a79731fa"} err="failed to get container status \"845c046f53f477aadd9728ceecd8603c1c4b9dc74ef60ccd2fd84b83a79731fa\": rpc error: code = NotFound desc = could not find container \"845c046f53f477aadd9728ceecd8603c1c4b9dc74ef60ccd2fd84b83a79731fa\": container with ID starting with 845c046f53f477aadd9728ceecd8603c1c4b9dc74ef60ccd2fd84b83a79731fa not found: ID does not exist" Oct 30 11:33:19.553356 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.553306 4289 scope.go:122] "RemoveContainer" containerID="7e321e52776566c38897e08cefdc4a28f01959033fed400e7fb51fa4d1c3b125" Oct 30 11:33:19.553620 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.553602 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e321e52776566c38897e08cefdc4a28f01959033fed400e7fb51fa4d1c3b125"} err="failed to get container status \"7e321e52776566c38897e08cefdc4a28f01959033fed400e7fb51fa4d1c3b125\": rpc error: code = NotFound desc = could not find container \"7e321e52776566c38897e08cefdc4a28f01959033fed400e7fb51fa4d1c3b125\": container with ID starting with 7e321e52776566c38897e08cefdc4a28f01959033fed400e7fb51fa4d1c3b125 not found: ID does not exist" Oct 30 11:33:19.553672 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.553623 4289 scope.go:122] "RemoveContainer" containerID="09347ec76edbb0b2c9acf02642b8e413dd84ef9bc7cd47461b8ed747cf3499f4" Oct 30 11:33:19.553672 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.553608 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r"] Oct 30 11:33:19.553877 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.553858 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09347ec76edbb0b2c9acf02642b8e413dd84ef9bc7cd47461b8ed747cf3499f4"} err="failed to get container status \"09347ec76edbb0b2c9acf02642b8e413dd84ef9bc7cd47461b8ed747cf3499f4\": rpc error: code = NotFound desc = could not find container \"09347ec76edbb0b2c9acf02642b8e413dd84ef9bc7cd47461b8ed747cf3499f4\": container with ID starting with 09347ec76edbb0b2c9acf02642b8e413dd84ef9bc7cd47461b8ed747cf3499f4 not found: ID does not exist" Oct 30 11:33:19.553925 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.553877 4289 scope.go:122] "RemoveContainer" containerID="845c046f53f477aadd9728ceecd8603c1c4b9dc74ef60ccd2fd84b83a79731fa" Oct 30 11:33:19.554091 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.554074 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"845c046f53f477aadd9728ceecd8603c1c4b9dc74ef60ccd2fd84b83a79731fa"} err="failed to get container status \"845c046f53f477aadd9728ceecd8603c1c4b9dc74ef60ccd2fd84b83a79731fa\": rpc error: code = NotFound desc = could not find container \"845c046f53f477aadd9728ceecd8603c1c4b9dc74ef60ccd2fd84b83a79731fa\": container with ID starting with 845c046f53f477aadd9728ceecd8603c1c4b9dc74ef60ccd2fd84b83a79731fa not found: ID does not exist" Oct 30 11:33:19.554130 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.554091 4289 scope.go:122] "RemoveContainer" containerID="7e321e52776566c38897e08cefdc4a28f01959033fed400e7fb51fa4d1c3b125" Oct 30 11:33:19.554291 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.554271 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e321e52776566c38897e08cefdc4a28f01959033fed400e7fb51fa4d1c3b125"} err="failed to get container status \"7e321e52776566c38897e08cefdc4a28f01959033fed400e7fb51fa4d1c3b125\": rpc error: code = NotFound desc = could not find container \"7e321e52776566c38897e08cefdc4a28f01959033fed400e7fb51fa4d1c3b125\": container with ID starting with 7e321e52776566c38897e08cefdc4a28f01959033fed400e7fb51fa4d1c3b125 not found: ID does not exist" Oct 30 11:33:19.554339 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.554293 4289 scope.go:122] "RemoveContainer" containerID="09347ec76edbb0b2c9acf02642b8e413dd84ef9bc7cd47461b8ed747cf3499f4" Oct 30 11:33:19.554809 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.554789 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09347ec76edbb0b2c9acf02642b8e413dd84ef9bc7cd47461b8ed747cf3499f4"} err="failed to get container status \"09347ec76edbb0b2c9acf02642b8e413dd84ef9bc7cd47461b8ed747cf3499f4\": rpc error: code = NotFound desc = could not find container \"09347ec76edbb0b2c9acf02642b8e413dd84ef9bc7cd47461b8ed747cf3499f4\": container with ID starting with 09347ec76edbb0b2c9acf02642b8e413dd84ef9bc7cd47461b8ed747cf3499f4 not found: ID does not exist" Oct 30 11:33:19.554864 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.554810 4289 scope.go:122] "RemoveContainer" containerID="78520c074aa268c8db0a53ee76495756244a44435d3c7800ed40c4de29f8baf7" Oct 30 11:33:19.559408 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.559389 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-b5v4r"] Oct 30 11:33:19.591804 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.591784 4289 scope.go:122] "RemoveContainer" containerID="f1837b2cdcc2402f8b98c1b33ec1d0f624b2024942a1964e405d945480d13140" Oct 30 11:33:19.596157 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.596135 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl"] Oct 30 11:33:19.599958 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.599939 4289 scope.go:122] "RemoveContainer" containerID="0cbabac544cdd981b0e7d1042b9f8cef758eb32e37c0b4475a62b406557596dd" Oct 30 11:33:19.600087 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.600068 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nrqbl"] Oct 30 11:33:19.609293 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.607626 4289 scope.go:122] "RemoveContainer" containerID="78520c074aa268c8db0a53ee76495756244a44435d3c7800ed40c4de29f8baf7" Oct 30 11:33:19.609293 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:19.607948 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78520c074aa268c8db0a53ee76495756244a44435d3c7800ed40c4de29f8baf7\": container with ID starting with 78520c074aa268c8db0a53ee76495756244a44435d3c7800ed40c4de29f8baf7 not found: ID does not exist" containerID="78520c074aa268c8db0a53ee76495756244a44435d3c7800ed40c4de29f8baf7" Oct 30 11:33:19.609293 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.607989 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78520c074aa268c8db0a53ee76495756244a44435d3c7800ed40c4de29f8baf7"} err="failed to get container status \"78520c074aa268c8db0a53ee76495756244a44435d3c7800ed40c4de29f8baf7\": rpc error: code = NotFound desc = could not find container \"78520c074aa268c8db0a53ee76495756244a44435d3c7800ed40c4de29f8baf7\": container with ID starting with 78520c074aa268c8db0a53ee76495756244a44435d3c7800ed40c4de29f8baf7 not found: ID does not exist" Oct 30 11:33:19.609293 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.608012 4289 scope.go:122] "RemoveContainer" containerID="f1837b2cdcc2402f8b98c1b33ec1d0f624b2024942a1964e405d945480d13140" Oct 30 11:33:19.609293 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:19.608273 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1837b2cdcc2402f8b98c1b33ec1d0f624b2024942a1964e405d945480d13140\": container with ID starting with f1837b2cdcc2402f8b98c1b33ec1d0f624b2024942a1964e405d945480d13140 not found: ID does not exist" containerID="f1837b2cdcc2402f8b98c1b33ec1d0f624b2024942a1964e405d945480d13140" Oct 30 11:33:19.609293 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.608294 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1837b2cdcc2402f8b98c1b33ec1d0f624b2024942a1964e405d945480d13140"} err="failed to get container status \"f1837b2cdcc2402f8b98c1b33ec1d0f624b2024942a1964e405d945480d13140\": rpc error: code = NotFound desc = could not find container \"f1837b2cdcc2402f8b98c1b33ec1d0f624b2024942a1964e405d945480d13140\": container with ID starting with f1837b2cdcc2402f8b98c1b33ec1d0f624b2024942a1964e405d945480d13140 not found: ID does not exist" Oct 30 11:33:19.609293 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.608312 4289 scope.go:122] "RemoveContainer" containerID="0cbabac544cdd981b0e7d1042b9f8cef758eb32e37c0b4475a62b406557596dd" Oct 30 11:33:19.609293 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:19.608888 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cbabac544cdd981b0e7d1042b9f8cef758eb32e37c0b4475a62b406557596dd\": container with ID starting with 0cbabac544cdd981b0e7d1042b9f8cef758eb32e37c0b4475a62b406557596dd not found: ID does not exist" containerID="0cbabac544cdd981b0e7d1042b9f8cef758eb32e37c0b4475a62b406557596dd" Oct 30 11:33:19.609293 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.608913 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cbabac544cdd981b0e7d1042b9f8cef758eb32e37c0b4475a62b406557596dd"} err="failed to get container status \"0cbabac544cdd981b0e7d1042b9f8cef758eb32e37c0b4475a62b406557596dd\": rpc error: code = NotFound desc = could not find container \"0cbabac544cdd981b0e7d1042b9f8cef758eb32e37c0b4475a62b406557596dd\": container with ID starting with 0cbabac544cdd981b0e7d1042b9f8cef758eb32e37c0b4475a62b406557596dd not found: ID does not exist" Oct 30 11:33:19.609293 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.608934 4289 scope.go:122] "RemoveContainer" containerID="78520c074aa268c8db0a53ee76495756244a44435d3c7800ed40c4de29f8baf7" Oct 30 11:33:19.609293 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.609197 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78520c074aa268c8db0a53ee76495756244a44435d3c7800ed40c4de29f8baf7"} err="failed to get container status \"78520c074aa268c8db0a53ee76495756244a44435d3c7800ed40c4de29f8baf7\": rpc error: code = NotFound desc = could not find container \"78520c074aa268c8db0a53ee76495756244a44435d3c7800ed40c4de29f8baf7\": container with ID starting with 78520c074aa268c8db0a53ee76495756244a44435d3c7800ed40c4de29f8baf7 not found: ID does not exist" Oct 30 11:33:19.609293 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.609217 4289 scope.go:122] "RemoveContainer" containerID="f1837b2cdcc2402f8b98c1b33ec1d0f624b2024942a1964e405d945480d13140" Oct 30 11:33:19.609848 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.609602 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1837b2cdcc2402f8b98c1b33ec1d0f624b2024942a1964e405d945480d13140"} err="failed to get container status \"f1837b2cdcc2402f8b98c1b33ec1d0f624b2024942a1964e405d945480d13140\": rpc error: code = NotFound desc = could not find container \"f1837b2cdcc2402f8b98c1b33ec1d0f624b2024942a1964e405d945480d13140\": container with ID starting with f1837b2cdcc2402f8b98c1b33ec1d0f624b2024942a1964e405d945480d13140 not found: ID does not exist" Oct 30 11:33:19.609848 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.609625 4289 scope.go:122] "RemoveContainer" containerID="0cbabac544cdd981b0e7d1042b9f8cef758eb32e37c0b4475a62b406557596dd" Oct 30 11:33:19.610160 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.610132 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cbabac544cdd981b0e7d1042b9f8cef758eb32e37c0b4475a62b406557596dd"} err="failed to get container status \"0cbabac544cdd981b0e7d1042b9f8cef758eb32e37c0b4475a62b406557596dd\": rpc error: code = NotFound desc = could not find container \"0cbabac544cdd981b0e7d1042b9f8cef758eb32e37c0b4475a62b406557596dd\": container with ID starting with 0cbabac544cdd981b0e7d1042b9f8cef758eb32e37c0b4475a62b406557596dd not found: ID does not exist" Oct 30 11:33:19.610214 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.610169 4289 scope.go:122] "RemoveContainer" containerID="78520c074aa268c8db0a53ee76495756244a44435d3c7800ed40c4de29f8baf7" Oct 30 11:33:19.610422 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.610403 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78520c074aa268c8db0a53ee76495756244a44435d3c7800ed40c4de29f8baf7"} err="failed to get container status \"78520c074aa268c8db0a53ee76495756244a44435d3c7800ed40c4de29f8baf7\": rpc error: code = NotFound desc = could not find container \"78520c074aa268c8db0a53ee76495756244a44435d3c7800ed40c4de29f8baf7\": container with ID starting with 78520c074aa268c8db0a53ee76495756244a44435d3c7800ed40c4de29f8baf7 not found: ID does not exist" Oct 30 11:33:19.610422 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.610424 4289 scope.go:122] "RemoveContainer" containerID="f1837b2cdcc2402f8b98c1b33ec1d0f624b2024942a1964e405d945480d13140" Oct 30 11:33:19.610711 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.610692 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1837b2cdcc2402f8b98c1b33ec1d0f624b2024942a1964e405d945480d13140"} err="failed to get container status \"f1837b2cdcc2402f8b98c1b33ec1d0f624b2024942a1964e405d945480d13140\": rpc error: code = NotFound desc = could not find container \"f1837b2cdcc2402f8b98c1b33ec1d0f624b2024942a1964e405d945480d13140\": container with ID starting with f1837b2cdcc2402f8b98c1b33ec1d0f624b2024942a1964e405d945480d13140 not found: ID does not exist" Oct 30 11:33:19.610762 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.610711 4289 scope.go:122] "RemoveContainer" containerID="0cbabac544cdd981b0e7d1042b9f8cef758eb32e37c0b4475a62b406557596dd" Oct 30 11:33:19.610948 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.610925 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cbabac544cdd981b0e7d1042b9f8cef758eb32e37c0b4475a62b406557596dd"} err="failed to get container status \"0cbabac544cdd981b0e7d1042b9f8cef758eb32e37c0b4475a62b406557596dd\": rpc error: code = NotFound desc = could not find container \"0cbabac544cdd981b0e7d1042b9f8cef758eb32e37c0b4475a62b406557596dd\": container with ID starting with 0cbabac544cdd981b0e7d1042b9f8cef758eb32e37c0b4475a62b406557596dd not found: ID does not exist" Oct 30 11:33:19.610993 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.610950 4289 scope.go:122] "RemoveContainer" containerID="a9618d3674faef0b3a8f8665665279803c9b25ebc50715fc7747f729ded56ba2" Oct 30 11:33:19.618022 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.618006 4289 scope.go:122] "RemoveContainer" containerID="b8b96c8f6fe7eee19ed7992ac90b36d3ae34829e2f1e294c51c30d4605e5a008" Oct 30 11:33:19.623807 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.623784 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62897926-e2e9-4408-b407-b1bdeb4ee0e9" path="/var/lib/kubelet/pods/62897926-e2e9-4408-b407-b1bdeb4ee0e9/volumes" Oct 30 11:33:19.624234 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.624216 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ade4e9f0-3694-4382-a2e9-5105763562b6" path="/var/lib/kubelet/pods/ade4e9f0-3694-4382-a2e9-5105763562b6/volumes" Oct 30 11:33:19.624969 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.624949 4289 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-tls-assets\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.624969 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.624967 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.625113 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.624976 4289 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.625113 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.624987 4289 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.625113 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.625014 4289 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") on node \"ip-10-0-137-242.ec2.internal\" " Oct 30 11:33:19.625113 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.625025 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7drp4\" (UniqueName: \"kubernetes.io/projected/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-kube-api-access-7drp4\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.625113 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.625034 4289 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.625113 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.625043 4289 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-config-out\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.625113 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.625051 4289 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-secret-prometheus-web-tls-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.625113 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.625059 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.625113 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.625068 4289 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-web-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.625113 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.625076 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.625113 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.625085 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/05d00ed1-5c02-43b7-a670-e5061c0cbbd5-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:19.625841 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.625828 4289 scope.go:122] "RemoveContainer" containerID="9484eb181fe0ae4005363c0db096ab13038ac17784c43d9a0daa85c69fa831c2" Oct 30 11:33:19.635972 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.635951 4289 scope.go:122] "RemoveContainer" containerID="d075553409a49f786fcbb732e70895b503bdf3f7f7342235515116df4bfadf99" Oct 30 11:33:19.638098 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.638083 4289 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-c799cc48-125a-421c-8fc6-8a4d017a7653" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48") on node "ip-10-0-137-242.ec2.internal" Oct 30 11:33:19.644026 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.644004 4289 scope.go:122] "RemoveContainer" containerID="a9618d3674faef0b3a8f8665665279803c9b25ebc50715fc7747f729ded56ba2" Oct 30 11:33:19.644303 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:19.644284 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9618d3674faef0b3a8f8665665279803c9b25ebc50715fc7747f729ded56ba2\": container with ID starting with a9618d3674faef0b3a8f8665665279803c9b25ebc50715fc7747f729ded56ba2 not found: ID does not exist" containerID="a9618d3674faef0b3a8f8665665279803c9b25ebc50715fc7747f729ded56ba2" Oct 30 11:33:19.644376 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.644317 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9618d3674faef0b3a8f8665665279803c9b25ebc50715fc7747f729ded56ba2"} err="failed to get container status \"a9618d3674faef0b3a8f8665665279803c9b25ebc50715fc7747f729ded56ba2\": rpc error: code = NotFound desc = could not find container \"a9618d3674faef0b3a8f8665665279803c9b25ebc50715fc7747f729ded56ba2\": container with ID starting with a9618d3674faef0b3a8f8665665279803c9b25ebc50715fc7747f729ded56ba2 not found: ID does not exist" Oct 30 11:33:19.644376 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.644341 4289 scope.go:122] "RemoveContainer" containerID="b8b96c8f6fe7eee19ed7992ac90b36d3ae34829e2f1e294c51c30d4605e5a008" Oct 30 11:33:19.644658 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:19.644632 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b96c8f6fe7eee19ed7992ac90b36d3ae34829e2f1e294c51c30d4605e5a008\": container with ID starting with b8b96c8f6fe7eee19ed7992ac90b36d3ae34829e2f1e294c51c30d4605e5a008 not found: ID does not exist" containerID="b8b96c8f6fe7eee19ed7992ac90b36d3ae34829e2f1e294c51c30d4605e5a008" Oct 30 11:33:19.644714 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.644664 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b96c8f6fe7eee19ed7992ac90b36d3ae34829e2f1e294c51c30d4605e5a008"} err="failed to get container status \"b8b96c8f6fe7eee19ed7992ac90b36d3ae34829e2f1e294c51c30d4605e5a008\": rpc error: code = NotFound desc = could not find container \"b8b96c8f6fe7eee19ed7992ac90b36d3ae34829e2f1e294c51c30d4605e5a008\": container with ID starting with b8b96c8f6fe7eee19ed7992ac90b36d3ae34829e2f1e294c51c30d4605e5a008 not found: ID does not exist" Oct 30 11:33:19.644714 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.644680 4289 scope.go:122] "RemoveContainer" containerID="9484eb181fe0ae4005363c0db096ab13038ac17784c43d9a0daa85c69fa831c2" Oct 30 11:33:19.644961 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:19.644943 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9484eb181fe0ae4005363c0db096ab13038ac17784c43d9a0daa85c69fa831c2\": container with ID starting with 9484eb181fe0ae4005363c0db096ab13038ac17784c43d9a0daa85c69fa831c2 not found: ID does not exist" containerID="9484eb181fe0ae4005363c0db096ab13038ac17784c43d9a0daa85c69fa831c2" Oct 30 11:33:19.645029 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.644966 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9484eb181fe0ae4005363c0db096ab13038ac17784c43d9a0daa85c69fa831c2"} err="failed to get container status \"9484eb181fe0ae4005363c0db096ab13038ac17784c43d9a0daa85c69fa831c2\": rpc error: code = NotFound desc = could not find container \"9484eb181fe0ae4005363c0db096ab13038ac17784c43d9a0daa85c69fa831c2\": container with ID starting with 9484eb181fe0ae4005363c0db096ab13038ac17784c43d9a0daa85c69fa831c2 not found: ID does not exist" Oct 30 11:33:19.645029 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.644979 4289 scope.go:122] "RemoveContainer" containerID="d075553409a49f786fcbb732e70895b503bdf3f7f7342235515116df4bfadf99" Oct 30 11:33:19.645218 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:19.645200 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d075553409a49f786fcbb732e70895b503bdf3f7f7342235515116df4bfadf99\": container with ID starting with d075553409a49f786fcbb732e70895b503bdf3f7f7342235515116df4bfadf99 not found: ID does not exist" containerID="d075553409a49f786fcbb732e70895b503bdf3f7f7342235515116df4bfadf99" Oct 30 11:33:19.645277 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.645225 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d075553409a49f786fcbb732e70895b503bdf3f7f7342235515116df4bfadf99"} err="failed to get container status \"d075553409a49f786fcbb732e70895b503bdf3f7f7342235515116df4bfadf99\": rpc error: code = NotFound desc = could not find container \"d075553409a49f786fcbb732e70895b503bdf3f7f7342235515116df4bfadf99\": container with ID starting with d075553409a49f786fcbb732e70895b503bdf3f7f7342235515116df4bfadf99 not found: ID does not exist" Oct 30 11:33:19.645277 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.645244 4289 scope.go:122] "RemoveContainer" containerID="a9618d3674faef0b3a8f8665665279803c9b25ebc50715fc7747f729ded56ba2" Oct 30 11:33:19.645577 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.645557 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9618d3674faef0b3a8f8665665279803c9b25ebc50715fc7747f729ded56ba2"} err="failed to get container status \"a9618d3674faef0b3a8f8665665279803c9b25ebc50715fc7747f729ded56ba2\": rpc error: code = NotFound desc = could not find container \"a9618d3674faef0b3a8f8665665279803c9b25ebc50715fc7747f729ded56ba2\": container with ID starting with a9618d3674faef0b3a8f8665665279803c9b25ebc50715fc7747f729ded56ba2 not found: ID does not exist" Oct 30 11:33:19.645654 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.645577 4289 scope.go:122] "RemoveContainer" containerID="b8b96c8f6fe7eee19ed7992ac90b36d3ae34829e2f1e294c51c30d4605e5a008" Oct 30 11:33:19.645827 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.645807 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b96c8f6fe7eee19ed7992ac90b36d3ae34829e2f1e294c51c30d4605e5a008"} err="failed to get container status \"b8b96c8f6fe7eee19ed7992ac90b36d3ae34829e2f1e294c51c30d4605e5a008\": rpc error: code = NotFound desc = could not find container \"b8b96c8f6fe7eee19ed7992ac90b36d3ae34829e2f1e294c51c30d4605e5a008\": container with ID starting with b8b96c8f6fe7eee19ed7992ac90b36d3ae34829e2f1e294c51c30d4605e5a008 not found: ID does not exist" Oct 30 11:33:19.645891 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.645827 4289 scope.go:122] "RemoveContainer" containerID="9484eb181fe0ae4005363c0db096ab13038ac17784c43d9a0daa85c69fa831c2" Oct 30 11:33:19.646045 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.646027 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9484eb181fe0ae4005363c0db096ab13038ac17784c43d9a0daa85c69fa831c2"} err="failed to get container status \"9484eb181fe0ae4005363c0db096ab13038ac17784c43d9a0daa85c69fa831c2\": rpc error: code = NotFound desc = could not find container \"9484eb181fe0ae4005363c0db096ab13038ac17784c43d9a0daa85c69fa831c2\": container with ID starting with 9484eb181fe0ae4005363c0db096ab13038ac17784c43d9a0daa85c69fa831c2 not found: ID does not exist" Oct 30 11:33:19.646109 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.646044 4289 scope.go:122] "RemoveContainer" containerID="d075553409a49f786fcbb732e70895b503bdf3f7f7342235515116df4bfadf99" Oct 30 11:33:19.646292 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.646274 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d075553409a49f786fcbb732e70895b503bdf3f7f7342235515116df4bfadf99"} err="failed to get container status \"d075553409a49f786fcbb732e70895b503bdf3f7f7342235515116df4bfadf99\": rpc error: code = NotFound desc = could not find container \"d075553409a49f786fcbb732e70895b503bdf3f7f7342235515116df4bfadf99\": container with ID starting with d075553409a49f786fcbb732e70895b503bdf3f7f7342235515116df4bfadf99 not found: ID does not exist" Oct 30 11:33:19.646360 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.646292 4289 scope.go:122] "RemoveContainer" containerID="a9618d3674faef0b3a8f8665665279803c9b25ebc50715fc7747f729ded56ba2" Oct 30 11:33:19.646572 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.646552 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9618d3674faef0b3a8f8665665279803c9b25ebc50715fc7747f729ded56ba2"} err="failed to get container status \"a9618d3674faef0b3a8f8665665279803c9b25ebc50715fc7747f729ded56ba2\": rpc error: code = NotFound desc = could not find container \"a9618d3674faef0b3a8f8665665279803c9b25ebc50715fc7747f729ded56ba2\": container with ID starting with a9618d3674faef0b3a8f8665665279803c9b25ebc50715fc7747f729ded56ba2 not found: ID does not exist" Oct 30 11:33:19.646572 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.646571 4289 scope.go:122] "RemoveContainer" containerID="b8b96c8f6fe7eee19ed7992ac90b36d3ae34829e2f1e294c51c30d4605e5a008" Oct 30 11:33:19.646793 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.646777 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b96c8f6fe7eee19ed7992ac90b36d3ae34829e2f1e294c51c30d4605e5a008"} err="failed to get container status \"b8b96c8f6fe7eee19ed7992ac90b36d3ae34829e2f1e294c51c30d4605e5a008\": rpc error: code = NotFound desc = could not find container \"b8b96c8f6fe7eee19ed7992ac90b36d3ae34829e2f1e294c51c30d4605e5a008\": container with ID starting with b8b96c8f6fe7eee19ed7992ac90b36d3ae34829e2f1e294c51c30d4605e5a008 not found: ID does not exist" Oct 30 11:33:19.646793 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.646792 4289 scope.go:122] "RemoveContainer" containerID="9484eb181fe0ae4005363c0db096ab13038ac17784c43d9a0daa85c69fa831c2" Oct 30 11:33:19.646991 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.646974 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9484eb181fe0ae4005363c0db096ab13038ac17784c43d9a0daa85c69fa831c2"} err="failed to get container status \"9484eb181fe0ae4005363c0db096ab13038ac17784c43d9a0daa85c69fa831c2\": rpc error: code = NotFound desc = could not find container \"9484eb181fe0ae4005363c0db096ab13038ac17784c43d9a0daa85c69fa831c2\": container with ID starting with 9484eb181fe0ae4005363c0db096ab13038ac17784c43d9a0daa85c69fa831c2 not found: ID does not exist" Oct 30 11:33:19.647066 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.646991 4289 scope.go:122] "RemoveContainer" containerID="d075553409a49f786fcbb732e70895b503bdf3f7f7342235515116df4bfadf99" Oct 30 11:33:19.647285 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.647252 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d075553409a49f786fcbb732e70895b503bdf3f7f7342235515116df4bfadf99"} err="failed to get container status \"d075553409a49f786fcbb732e70895b503bdf3f7f7342235515116df4bfadf99\": rpc error: code = NotFound desc = could not find container \"d075553409a49f786fcbb732e70895b503bdf3f7f7342235515116df4bfadf99\": container with ID starting with d075553409a49f786fcbb732e70895b503bdf3f7f7342235515116df4bfadf99 not found: ID does not exist" Oct 30 11:33:19.650162 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.650140 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl"] Oct 30 11:33:19.653880 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.653862 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-bv6wl"] Oct 30 11:33:19.688717 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.688689 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w"] Oct 30 11:33:19.690651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.690622 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-56n4w"] Oct 30 11:33:19.726061 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.726030 4289 reconciler_common.go:299] "Volume detached for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"csi-ea06c66241a0a003a5daff5d9fc602f40e53768b00b104c5e4118b92d558c090\"" Oct 30 11:33:19.865069 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.864976 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:33:19.867901 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.867874 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:33:19.899277 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:19.899254 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:20.028146 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.028115 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-web-config\" (UniqueName: \"kubernetes.io/secret/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-web-config\") pod \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " Oct 30 11:33:20.028328 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.028168 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-config-volume\" (UniqueName: \"kubernetes.io/secret/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-config-volume\") pod \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " Oct 30 11:33:20.028328 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.028203 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-cluster-tls-config\") pod \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " Oct 30 11:33:20.028328 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.028235 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-config-out\") pod \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " Oct 30 11:33:20.028328 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.028275 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-alertmanager-data-science-monitoringstack-db\") pod \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " Oct 30 11:33:20.028328 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.028292 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-tls-assets\" (UniqueName: \"kubernetes.io/projected/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-tls-assets\") pod \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " Oct 30 11:33:20.028635 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.028353 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-kube-api-access-bb258\" (UniqueName: \"kubernetes.io/projected/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-kube-api-access-bb258\") pod \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\" (UID: \"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e\") " Oct 30 11:33:20.028697 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.028674 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-alertmanager-data-science-monitoringstack-db" pod "ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e" (UID: "ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:33:20.031006 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.030816 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-config-volume" pod "ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e" (UID: "ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:33:20.031006 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.030891 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-config-out" pod "ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e" (UID: "ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:33:20.031006 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.030976 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-tls-assets" pod "ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e" (UID: "ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:33:20.031006 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.030985 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-kube-api-access-bb258" pod "ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e" (UID: "ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e"). InnerVolumeSpecName "kube-api-access-bb258". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:33:20.035859 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.035828 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-cluster-tls-config" pod "ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e" (UID: "ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:33:20.042657 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.042638 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-web-config" pod "ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e" (UID: "ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:33:20.129721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.129644 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb258\" (UniqueName: \"kubernetes.io/projected/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-kube-api-access-bb258\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:20.129721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.129669 4289 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-web-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:20.129721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.129678 4289 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:20.129721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.129686 4289 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-cluster-tls-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:20.129721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.129694 4289 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-config-out\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:20.129721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.129703 4289 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:20.129721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.129714 4289 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e-tls-assets\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:20.142513 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142484 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 11:33:20.142803 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142792 4289 cpu_manager.go:414] "containerMap: removing container" podUID="953398bd-24a9-4221-88bd-601a0796cf64" containerName="kube-rbac-proxy" Oct 30 11:33:20.142847 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142805 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="953398bd-24a9-4221-88bd-601a0796cf64" containerName="kube-rbac-proxy" Oct 30 11:33:20.142847 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142815 4289 cpu_manager.go:414] "containerMap: removing container" podUID="ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e" containerName="alertmanager" Oct 30 11:33:20.142847 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142820 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e" containerName="alertmanager" Oct 30 11:33:20.142847 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142834 4289 cpu_manager.go:414] "containerMap: removing container" podUID="05d00ed1-5c02-43b7-a670-e5061c0cbbd5" containerName="thanos-sidecar" Oct 30 11:33:20.142847 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142840 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d00ed1-5c02-43b7-a670-e5061c0cbbd5" containerName="thanos-sidecar" Oct 30 11:33:20.142847 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142846 4289 cpu_manager.go:414] "containerMap: removing container" podUID="ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e" containerName="config-reloader" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142853 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e" containerName="config-reloader" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142864 4289 cpu_manager.go:414] "containerMap: removing container" podUID="e9599347-1676-48b3-bf20-5ba203487718" containerName="prom-label-proxy" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142874 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9599347-1676-48b3-bf20-5ba203487718" containerName="prom-label-proxy" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142883 4289 cpu_manager.go:414] "containerMap: removing container" podUID="e9599347-1676-48b3-bf20-5ba203487718" containerName="kube-rbac-proxy" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142887 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9599347-1676-48b3-bf20-5ba203487718" containerName="kube-rbac-proxy" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142892 4289 cpu_manager.go:414] "containerMap: removing container" podUID="ade4e9f0-3694-4382-a2e9-5105763562b6" containerName="korrel8r" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142896 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade4e9f0-3694-4382-a2e9-5105763562b6" containerName="korrel8r" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142904 4289 cpu_manager.go:414] "containerMap: removing container" podUID="62897926-e2e9-4408-b407-b1bdeb4ee0e9" containerName="kube-rbac-proxy" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142913 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62897926-e2e9-4408-b407-b1bdeb4ee0e9" containerName="kube-rbac-proxy" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142920 4289 cpu_manager.go:414] "containerMap: removing container" podUID="953398bd-24a9-4221-88bd-601a0796cf64" containerName="method-gate" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142924 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="953398bd-24a9-4221-88bd-601a0796cf64" containerName="method-gate" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142930 4289 cpu_manager.go:414] "containerMap: removing container" podUID="05d00ed1-5c02-43b7-a670-e5061c0cbbd5" containerName="config-reloader" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142934 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d00ed1-5c02-43b7-a670-e5061c0cbbd5" containerName="config-reloader" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142942 4289 cpu_manager.go:414] "containerMap: removing container" podUID="e9599347-1676-48b3-bf20-5ba203487718" containerName="method-gate" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142947 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9599347-1676-48b3-bf20-5ba203487718" containerName="method-gate" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142952 4289 cpu_manager.go:414] "containerMap: removing container" podUID="ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e" containerName="init-config-reloader" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142956 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e" containerName="init-config-reloader" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142961 4289 cpu_manager.go:414] "containerMap: removing container" podUID="05d00ed1-5c02-43b7-a670-e5061c0cbbd5" containerName="prometheus" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142965 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d00ed1-5c02-43b7-a670-e5061c0cbbd5" containerName="prometheus" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142973 4289 cpu_manager.go:414] "containerMap: removing container" podUID="05d00ed1-5c02-43b7-a670-e5061c0cbbd5" containerName="init-config-reloader" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142977 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d00ed1-5c02-43b7-a670-e5061c0cbbd5" containerName="init-config-reloader" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142985 4289 cpu_manager.go:414] "containerMap: removing container" podUID="a69e1e25-95cf-4907-81af-69c0c7b545a4" containerName="otc-container" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142988 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a69e1e25-95cf-4907-81af-69c0c7b545a4" containerName="otc-container" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142994 4289 cpu_manager.go:414] "containerMap: removing container" podUID="953398bd-24a9-4221-88bd-601a0796cf64" containerName="prom-label-proxy" Oct 30 11:33:20.143023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.142999 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="953398bd-24a9-4221-88bd-601a0796cf64" containerName="prom-label-proxy" Oct 30 11:33:20.143730 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.143109 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9599347-1676-48b3-bf20-5ba203487718" containerName="kube-rbac-proxy" Oct 30 11:33:20.143730 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.143119 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade4e9f0-3694-4382-a2e9-5105763562b6" containerName="korrel8r" Oct 30 11:33:20.143730 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.143125 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="62897926-e2e9-4408-b407-b1bdeb4ee0e9" containerName="kube-rbac-proxy" Oct 30 11:33:20.143730 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.143131 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="953398bd-24a9-4221-88bd-601a0796cf64" containerName="method-gate" Oct 30 11:33:20.143730 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.143137 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="05d00ed1-5c02-43b7-a670-e5061c0cbbd5" containerName="config-reloader" Oct 30 11:33:20.143730 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.143142 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9599347-1676-48b3-bf20-5ba203487718" containerName="method-gate" Oct 30 11:33:20.143730 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.143147 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="05d00ed1-5c02-43b7-a670-e5061c0cbbd5" containerName="prometheus" Oct 30 11:33:20.143730 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.143153 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="a69e1e25-95cf-4907-81af-69c0c7b545a4" containerName="otc-container" Oct 30 11:33:20.143730 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.143158 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="953398bd-24a9-4221-88bd-601a0796cf64" containerName="prom-label-proxy" Oct 30 11:33:20.143730 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.143162 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="953398bd-24a9-4221-88bd-601a0796cf64" containerName="kube-rbac-proxy" Oct 30 11:33:20.143730 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.143172 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e" containerName="alertmanager" Oct 30 11:33:20.143730 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.143178 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="05d00ed1-5c02-43b7-a670-e5061c0cbbd5" containerName="thanos-sidecar" Oct 30 11:33:20.143730 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.143182 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e" containerName="config-reloader" Oct 30 11:33:20.143730 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.143189 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9599347-1676-48b3-bf20-5ba203487718" containerName="prom-label-proxy" Oct 30 11:33:20.146260 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.146244 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:33:20.163813 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.163781 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 11:33:20.164943 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.164924 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 11:33:20.166673 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.166650 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-2cltz\"" Oct 30 11:33:20.167091 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.167069 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-e4cadd38\"" Oct 30 11:33:20.167208 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.167164 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 11:33:20.169917 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.169895 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 11:33:20.176840 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.176817 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 11:33:20.177093 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:20.177072 4289 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[data-science-collector-ta-client-cert kube-api-access-gps9m monitoring-tls-certs otc-internal tls-certs], unattached volumes=[], failed to process volumes=[]: context canceled" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="1dede02a-327c-41e0-b11a-073bf41832d4" Oct 30 11:33:20.230645 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.230610 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1dede02a-327c-41e0-b11a-073bf41832d4-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"1dede02a-327c-41e0-b11a-073bf41832d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:33:20.230824 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.230731 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gps9m\" (UniqueName: \"kubernetes.io/projected/1dede02a-327c-41e0-b11a-073bf41832d4-kube-api-access-gps9m\") pod \"data-science-collector-collector-0\" (UID: \"1dede02a-327c-41e0-b11a-073bf41832d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:33:20.230824 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.230763 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dede02a-327c-41e0-b11a-073bf41832d4-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1dede02a-327c-41e0-b11a-073bf41832d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:33:20.230943 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.230831 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1dede02a-327c-41e0-b11a-073bf41832d4-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1dede02a-327c-41e0-b11a-073bf41832d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:33:20.230943 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.230873 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1dede02a-327c-41e0-b11a-073bf41832d4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"1dede02a-327c-41e0-b11a-073bf41832d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:33:20.267351 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.267308 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:20.337839 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.337794 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gps9m\" (UniqueName: \"kubernetes.io/projected/1dede02a-327c-41e0-b11a-073bf41832d4-kube-api-access-gps9m\") pod \"data-science-collector-collector-0\" (UID: \"1dede02a-327c-41e0-b11a-073bf41832d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:33:20.338100 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.338077 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dede02a-327c-41e0-b11a-073bf41832d4-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1dede02a-327c-41e0-b11a-073bf41832d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:33:20.338207 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.338123 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1dede02a-327c-41e0-b11a-073bf41832d4-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1dede02a-327c-41e0-b11a-073bf41832d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:33:20.338207 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.338165 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1dede02a-327c-41e0-b11a-073bf41832d4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"1dede02a-327c-41e0-b11a-073bf41832d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:33:20.338514 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.338219 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1dede02a-327c-41e0-b11a-073bf41832d4-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"1dede02a-327c-41e0-b11a-073bf41832d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:33:20.338607 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:20.338567 4289 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-collector-collector-e4cadd38: configmap "data-science-collector-collector-e4cadd38" not found Oct 30 11:33:20.338661 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:20.338636 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1dede02a-327c-41e0-b11a-073bf41832d4-otc-internal podName:1dede02a-327c-41e0-b11a-073bf41832d4 nodeName:}" failed. No retries permitted until 2026-10-30 11:33:20.838613632 +0000 UTC m=+2891.339993111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/1dede02a-327c-41e0-b11a-073bf41832d4-otc-internal") pod "data-science-collector-collector-0" (UID: "1dede02a-327c-41e0-b11a-073bf41832d4") : configmap "data-science-collector-collector-e4cadd38" not found Oct 30 11:33:20.341094 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.341063 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1dede02a-327c-41e0-b11a-073bf41832d4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"1dede02a-327c-41e0-b11a-073bf41832d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:33:20.341222 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.341111 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dede02a-327c-41e0-b11a-073bf41832d4-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1dede02a-327c-41e0-b11a-073bf41832d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:33:20.341222 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.341132 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1dede02a-327c-41e0-b11a-073bf41832d4-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1dede02a-327c-41e0-b11a-073bf41832d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:33:20.367103 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:20.367075 4289 projected.go:196] Error preparing data for projected volume kube-api-access-gps9m for pod redhat-ods-monitoring/data-science-collector-collector-0: failed to fetch token: serviceaccounts "data-science-collector-collector" not found Oct 30 11:33:20.367227 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:20.367136 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1dede02a-327c-41e0-b11a-073bf41832d4-kube-api-access-gps9m podName:1dede02a-327c-41e0-b11a-073bf41832d4 nodeName:}" failed. No retries permitted until 2026-10-30 11:33:20.867121508 +0000 UTC m=+2891.368500975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gps9m" (UniqueName: "kubernetes.io/projected/1dede02a-327c-41e0-b11a-073bf41832d4-kube-api-access-gps9m") pod "data-science-collector-collector-0" (UID: "1dede02a-327c-41e0-b11a-073bf41832d4") : failed to fetch token: serviceaccounts "data-science-collector-collector" not found Oct 30 11:33:20.435074 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.434985 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:20.530952 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.530909 4289 generic.go:356] "Generic (PLEG): container finished" podID="ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e" containerID="6092255c5960182ac4874689d074fb9a3d9f442d5d54f719f1c7615b45ec3f37" exitCode=0 Oct 30 11:33:20.531412 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.530993 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:20.531412 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.530995 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e","Type":"ContainerDied","Data":"6092255c5960182ac4874689d074fb9a3d9f442d5d54f719f1c7615b45ec3f37"} Oct 30 11:33:20.531412 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.531092 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e","Type":"ContainerDied","Data":"5496d82bcc77e68cf1107846c89a90041b607575f0eddb74ff49d7e8adb89f73"} Oct 30 11:33:20.531412 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.531133 4289 scope.go:122] "RemoveContainer" containerID="685156e56f8b61506cf80a78eeb80a9e72787c001021e6160195ce16adc94300" Oct 30 11:33:20.532194 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.532167 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:33:20.539330 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.539307 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:33:20.540631 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.540613 4289 scope.go:122] "RemoveContainer" containerID="6092255c5960182ac4874689d074fb9a3d9f442d5d54f719f1c7615b45ec3f37" Oct 30 11:33:20.548476 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.548453 4289 scope.go:122] "RemoveContainer" containerID="4fabd20cba075c0d4650b9abdaea12702060682fa315f9d440d76709d16afba9" Oct 30 11:33:20.555542 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.555519 4289 scope.go:122] "RemoveContainer" containerID="685156e56f8b61506cf80a78eeb80a9e72787c001021e6160195ce16adc94300" Oct 30 11:33:20.555824 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:20.555802 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"685156e56f8b61506cf80a78eeb80a9e72787c001021e6160195ce16adc94300\": container with ID starting with 685156e56f8b61506cf80a78eeb80a9e72787c001021e6160195ce16adc94300 not found: ID does not exist" containerID="685156e56f8b61506cf80a78eeb80a9e72787c001021e6160195ce16adc94300" Oct 30 11:33:20.555907 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.555840 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"685156e56f8b61506cf80a78eeb80a9e72787c001021e6160195ce16adc94300"} err="failed to get container status \"685156e56f8b61506cf80a78eeb80a9e72787c001021e6160195ce16adc94300\": rpc error: code = NotFound desc = could not find container \"685156e56f8b61506cf80a78eeb80a9e72787c001021e6160195ce16adc94300\": container with ID starting with 685156e56f8b61506cf80a78eeb80a9e72787c001021e6160195ce16adc94300 not found: ID does not exist" Oct 30 11:33:20.555907 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.555858 4289 scope.go:122] "RemoveContainer" containerID="6092255c5960182ac4874689d074fb9a3d9f442d5d54f719f1c7615b45ec3f37" Oct 30 11:33:20.556119 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:20.556105 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6092255c5960182ac4874689d074fb9a3d9f442d5d54f719f1c7615b45ec3f37\": container with ID starting with 6092255c5960182ac4874689d074fb9a3d9f442d5d54f719f1c7615b45ec3f37 not found: ID does not exist" containerID="6092255c5960182ac4874689d074fb9a3d9f442d5d54f719f1c7615b45ec3f37" Oct 30 11:33:20.556168 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.556122 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6092255c5960182ac4874689d074fb9a3d9f442d5d54f719f1c7615b45ec3f37"} err="failed to get container status \"6092255c5960182ac4874689d074fb9a3d9f442d5d54f719f1c7615b45ec3f37\": rpc error: code = NotFound desc = could not find container \"6092255c5960182ac4874689d074fb9a3d9f442d5d54f719f1c7615b45ec3f37\": container with ID starting with 6092255c5960182ac4874689d074fb9a3d9f442d5d54f719f1c7615b45ec3f37 not found: ID does not exist" Oct 30 11:33:20.556168 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.556134 4289 scope.go:122] "RemoveContainer" containerID="4fabd20cba075c0d4650b9abdaea12702060682fa315f9d440d76709d16afba9" Oct 30 11:33:20.556386 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:20.556365 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fabd20cba075c0d4650b9abdaea12702060682fa315f9d440d76709d16afba9\": container with ID starting with 4fabd20cba075c0d4650b9abdaea12702060682fa315f9d440d76709d16afba9 not found: ID does not exist" containerID="4fabd20cba075c0d4650b9abdaea12702060682fa315f9d440d76709d16afba9" Oct 30 11:33:20.556450 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.556394 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fabd20cba075c0d4650b9abdaea12702060682fa315f9d440d76709d16afba9"} err="failed to get container status \"4fabd20cba075c0d4650b9abdaea12702060682fa315f9d440d76709d16afba9\": rpc error: code = NotFound desc = could not find container \"4fabd20cba075c0d4650b9abdaea12702060682fa315f9d440d76709d16afba9\": container with ID starting with 4fabd20cba075c0d4650b9abdaea12702060682fa315f9d440d76709d16afba9 not found: ID does not exist" Oct 30 11:33:20.625274 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.625239 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:33:20.629117 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.629090 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:33:20.640128 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.640104 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dede02a-327c-41e0-b11a-073bf41832d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/1dede02a-327c-41e0-b11a-073bf41832d4-tls-certs\") pod \"1dede02a-327c-41e0-b11a-073bf41832d4\" (UID: \"1dede02a-327c-41e0-b11a-073bf41832d4\") " Oct 30 11:33:20.640251 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.640145 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dede02a-327c-41e0-b11a-073bf41832d4-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1dede02a-327c-41e0-b11a-073bf41832d4-data-science-collector-ta-client-cert\") pod \"1dede02a-327c-41e0-b11a-073bf41832d4\" (UID: \"1dede02a-327c-41e0-b11a-073bf41832d4\") " Oct 30 11:33:20.640251 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.640169 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dede02a-327c-41e0-b11a-073bf41832d4-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1dede02a-327c-41e0-b11a-073bf41832d4-monitoring-tls-certs\") pod \"1dede02a-327c-41e0-b11a-073bf41832d4\" (UID: \"1dede02a-327c-41e0-b11a-073bf41832d4\") " Oct 30 11:33:20.642259 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.642235 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dede02a-327c-41e0-b11a-073bf41832d4-data-science-collector-ta-client-cert" pod "1dede02a-327c-41e0-b11a-073bf41832d4" (UID: "1dede02a-327c-41e0-b11a-073bf41832d4"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:33:20.642378 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.642256 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dede02a-327c-41e0-b11a-073bf41832d4-tls-certs" pod "1dede02a-327c-41e0-b11a-073bf41832d4" (UID: "1dede02a-327c-41e0-b11a-073bf41832d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:33:20.642378 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.642279 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dede02a-327c-41e0-b11a-073bf41832d4-monitoring-tls-certs" pod "1dede02a-327c-41e0-b11a-073bf41832d4" (UID: "1dede02a-327c-41e0-b11a-073bf41832d4"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:33:20.741513 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.741482 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dede02a-327c-41e0-b11a-073bf41832d4-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:20.741513 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.741510 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1dede02a-327c-41e0-b11a-073bf41832d4-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:20.741774 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.741523 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1dede02a-327c-41e0-b11a-073bf41832d4-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:20.842600 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.842558 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1dede02a-327c-41e0-b11a-073bf41832d4-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"1dede02a-327c-41e0-b11a-073bf41832d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:33:20.842792 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:20.842670 4289 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-collector-collector-e4cadd38: configmap "data-science-collector-collector-e4cadd38" not found Oct 30 11:33:20.842792 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:20.842749 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1dede02a-327c-41e0-b11a-073bf41832d4-otc-internal podName:1dede02a-327c-41e0-b11a-073bf41832d4 nodeName:}" failed. No retries permitted until 2026-10-30 11:33:21.842727739 +0000 UTC m=+2892.344107207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/1dede02a-327c-41e0-b11a-073bf41832d4-otc-internal") pod "data-science-collector-collector-0" (UID: "1dede02a-327c-41e0-b11a-073bf41832d4") : configmap "data-science-collector-collector-e4cadd38" not found Oct 30 11:33:20.944039 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:20.944004 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gps9m\" (UniqueName: \"kubernetes.io/projected/1dede02a-327c-41e0-b11a-073bf41832d4-kube-api-access-gps9m\") pod \"data-science-collector-collector-0\" (UID: \"1dede02a-327c-41e0-b11a-073bf41832d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:33:20.959687 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:20.959662 4289 projected.go:196] Error preparing data for projected volume kube-api-access-gps9m for pod redhat-ods-monitoring/data-science-collector-collector-0: failed to fetch token: serviceaccounts "data-science-collector-collector" not found Oct 30 11:33:20.959813 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:20.959723 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1dede02a-327c-41e0-b11a-073bf41832d4-kube-api-access-gps9m podName:1dede02a-327c-41e0-b11a-073bf41832d4 nodeName:}" failed. No retries permitted until 2026-10-30 11:33:21.959705986 +0000 UTC m=+2892.461085464 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gps9m" (UniqueName: "kubernetes.io/projected/1dede02a-327c-41e0-b11a-073bf41832d4-kube-api-access-gps9m") pod "data-science-collector-collector-0" (UID: "1dede02a-327c-41e0-b11a-073bf41832d4") : failed to fetch token: serviceaccounts "data-science-collector-collector" not found Oct 30 11:33:21.126200 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:21.126111 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:21.307892 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:21.307860 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:21.538215 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:21.538190 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:33:21.614618 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:21.614586 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 11:33:21.624582 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:21.624553 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05d00ed1-5c02-43b7-a670-e5061c0cbbd5" path="/var/lib/kubelet/pods/05d00ed1-5c02-43b7-a670-e5061c0cbbd5/volumes" Oct 30 11:33:21.625060 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:21.625047 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="953398bd-24a9-4221-88bd-601a0796cf64" path="/var/lib/kubelet/pods/953398bd-24a9-4221-88bd-601a0796cf64/volumes" Oct 30 11:33:21.625748 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:21.625730 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e" path="/var/lib/kubelet/pods/ca51d32e-4be1-4a5a-b58f-c1fcd84cf86e/volumes" Oct 30 11:33:21.626151 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:21.626139 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9599347-1676-48b3-bf20-5ba203487718" path="/var/lib/kubelet/pods/e9599347-1676-48b3-bf20-5ba203487718/volumes" Oct 30 11:33:21.626473 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:21.626460 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 11:33:21.749723 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:21.749694 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gps9m\" (UniqueName: \"kubernetes.io/projected/1dede02a-327c-41e0-b11a-073bf41832d4-kube-api-access-gps9m\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:21.749723 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:21.749723 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1dede02a-327c-41e0-b11a-073bf41832d4-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:23.623579 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:23.623547 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dede02a-327c-41e0-b11a-073bf41832d4" path="/var/lib/kubelet/pods/1dede02a-327c-41e0-b11a-073bf41832d4/volumes" Oct 30 11:33:26.197310 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.197268 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:26.258604 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:26.258570 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:33:26.361344 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.361313 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:26.452544 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.452465 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7"] Oct 30 11:33:26.457048 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.457031 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" Oct 30 11:33:26.472230 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.472196 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2"] Oct 30 11:33:26.477587 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.477562 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" Oct 30 11:33:26.482416 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.482378 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-dl8jw\"" Oct 30 11:33:26.482557 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.482537 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 11:33:26.482622 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.482593 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 11:33:26.485889 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.485867 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 11:33:26.485994 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.485894 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 11:33:26.495951 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.495929 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 11:33:26.496155 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.496138 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 11:33:26.497594 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.497573 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7"] Oct 30 11:33:26.498644 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.498627 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2"] Oct 30 11:33:26.507489 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.507463 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48"] Oct 30 11:33:26.511403 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.511383 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" Oct 30 11:33:26.534050 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.534026 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48"] Oct 30 11:33:26.537639 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.537614 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:26.556241 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.556213 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg"] Oct 30 11:33:26.559703 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.559685 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg" Oct 30 11:33:26.573344 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.573325 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 11:33:26.573529 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.573398 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 11:33:26.573529 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.573504 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-k428p\"" Oct 30 11:33:26.573529 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.573514 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-v89j7\"" Oct 30 11:33:26.576490 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.576468 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 11:33:26.576490 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.576479 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 11:33:26.576788 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.576493 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 11:33:26.576788 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.576542 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 11:33:26.578272 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.578248 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg"] Oct 30 11:33:26.592442 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.592404 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/86e3cacb-a113-4182-b955-fd45c0e77190-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8v5d7\" (UID: \"86e3cacb-a113-4182-b955-fd45c0e77190\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" Oct 30 11:33:26.592540 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.592454 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/17a81880-2dca-4878-bf5a-6f7185f4451e-config\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2\" (UID: \"17a81880-2dca-4878-bf5a-6f7185f4451e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" Oct 30 11:33:26.592594 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.592548 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/86e3cacb-a113-4182-b955-fd45c0e77190-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8v5d7\" (UID: \"86e3cacb-a113-4182-b955-fd45c0e77190\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" Oct 30 11:33:26.592594 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.592570 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/17a81880-2dca-4878-bf5a-6f7185f4451e-client-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2\" (UID: \"17a81880-2dca-4878-bf5a-6f7185f4451e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" Oct 30 11:33:26.592594 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.592589 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/86e3cacb-a113-4182-b955-fd45c0e77190-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8v5d7\" (UID: \"86e3cacb-a113-4182-b955-fd45c0e77190\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" Oct 30 11:33:26.592690 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.592675 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86e3cacb-a113-4182-b955-fd45c0e77190-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8v5d7\" (UID: \"86e3cacb-a113-4182-b955-fd45c0e77190\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" Oct 30 11:33:26.592732 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.592716 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqx2r\" (UniqueName: \"kubernetes.io/projected/86e3cacb-a113-4182-b955-fd45c0e77190-kube-api-access-lqx2r\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8v5d7\" (UID: \"86e3cacb-a113-4182-b955-fd45c0e77190\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" Oct 30 11:33:26.592793 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.592772 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/17a81880-2dca-4878-bf5a-6f7185f4451e-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2\" (UID: \"17a81880-2dca-4878-bf5a-6f7185f4451e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" Oct 30 11:33:26.592829 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.592806 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/17a81880-2dca-4878-bf5a-6f7185f4451e-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2\" (UID: \"17a81880-2dca-4878-bf5a-6f7185f4451e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" Oct 30 11:33:26.592860 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.592829 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lv9g\" (UniqueName: \"kubernetes.io/projected/17a81880-2dca-4878-bf5a-6f7185f4451e-kube-api-access-6lv9g\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2\" (UID: \"17a81880-2dca-4878-bf5a-6f7185f4451e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" Oct 30 11:33:26.673501 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.673466 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-nwmbn\"" Oct 30 11:33:26.673669 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.673536 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 11:33:26.693731 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.693710 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/17a81880-2dca-4878-bf5a-6f7185f4451e-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2\" (UID: \"17a81880-2dca-4878-bf5a-6f7185f4451e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" Oct 30 11:33:26.693841 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.693740 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2835390b-006c-4fc4-81fc-3ab4952f4dc6-config\") pod \"data-science-thanos-querier-proxy-7b56895dd5-hmd48\" (UID: \"2835390b-006c-4fc4-81fc-3ab4952f4dc6\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" Oct 30 11:33:26.693841 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.693766 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpgwh\" (UniqueName: \"kubernetes.io/projected/2835390b-006c-4fc4-81fc-3ab4952f4dc6-kube-api-access-qpgwh\") pod \"data-science-thanos-querier-proxy-7b56895dd5-hmd48\" (UID: \"2835390b-006c-4fc4-81fc-3ab4952f4dc6\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" Oct 30 11:33:26.693841 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.693791 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/17a81880-2dca-4878-bf5a-6f7185f4451e-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2\" (UID: \"17a81880-2dca-4878-bf5a-6f7185f4451e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" Oct 30 11:33:26.693998 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.693883 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lv9g\" (UniqueName: \"kubernetes.io/projected/17a81880-2dca-4878-bf5a-6f7185f4451e-kube-api-access-6lv9g\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2\" (UID: \"17a81880-2dca-4878-bf5a-6f7185f4451e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" Oct 30 11:33:26.693998 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.693917 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/86e3cacb-a113-4182-b955-fd45c0e77190-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8v5d7\" (UID: \"86e3cacb-a113-4182-b955-fd45c0e77190\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" Oct 30 11:33:26.693998 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.693940 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/17a81880-2dca-4878-bf5a-6f7185f4451e-config\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2\" (UID: \"17a81880-2dca-4878-bf5a-6f7185f4451e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" Oct 30 11:33:26.693998 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.693958 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/098dce60-e115-4404-8e5f-43809c74e6b6-config\") pod \"korrel8r-5cbcc89b8f-xvlpg\" (UID: \"098dce60-e115-4404-8e5f-43809c74e6b6\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg" Oct 30 11:33:26.693998 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:26.693967 4289 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-prometheus-namespace-proxy-tls: secret "data-science-prometheus-namespace-proxy-tls" not found Oct 30 11:33:26.694231 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:26.694033 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a81880-2dca-4878-bf5a-6f7185f4451e-tls podName:17a81880-2dca-4878-bf5a-6f7185f4451e nodeName:}" failed. No retries permitted until 2026-10-30 11:33:27.194010208 +0000 UTC m=+2897.695389697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/17a81880-2dca-4878-bf5a-6f7185f4451e-tls") pod "data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" (UID: "17a81880-2dca-4878-bf5a-6f7185f4451e") : secret "data-science-prometheus-namespace-proxy-tls" not found Oct 30 11:33:26.694231 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.694077 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/86e3cacb-a113-4182-b955-fd45c0e77190-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8v5d7\" (UID: \"86e3cacb-a113-4182-b955-fd45c0e77190\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" Oct 30 11:33:26.694231 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.694103 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/17a81880-2dca-4878-bf5a-6f7185f4451e-client-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2\" (UID: \"17a81880-2dca-4878-bf5a-6f7185f4451e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" Oct 30 11:33:26.694231 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.694129 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/86e3cacb-a113-4182-b955-fd45c0e77190-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8v5d7\" (UID: \"86e3cacb-a113-4182-b955-fd45c0e77190\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" Oct 30 11:33:26.694231 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.694171 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/098dce60-e115-4404-8e5f-43809c74e6b6-tls\") pod \"korrel8r-5cbcc89b8f-xvlpg\" (UID: \"098dce60-e115-4404-8e5f-43809c74e6b6\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg" Oct 30 11:33:26.694231 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.694201 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2835390b-006c-4fc4-81fc-3ab4952f4dc6-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-hmd48\" (UID: \"2835390b-006c-4fc4-81fc-3ab4952f4dc6\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" Oct 30 11:33:26.694231 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.694232 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86e3cacb-a113-4182-b955-fd45c0e77190-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8v5d7\" (UID: \"86e3cacb-a113-4182-b955-fd45c0e77190\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" Oct 30 11:33:26.694630 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.694256 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mprxd\" (UniqueName: \"kubernetes.io/projected/098dce60-e115-4404-8e5f-43809c74e6b6-kube-api-access-mprxd\") pod \"korrel8r-5cbcc89b8f-xvlpg\" (UID: \"098dce60-e115-4404-8e5f-43809c74e6b6\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg" Oct 30 11:33:26.694630 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.694293 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqx2r\" (UniqueName: \"kubernetes.io/projected/86e3cacb-a113-4182-b955-fd45c0e77190-kube-api-access-lqx2r\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8v5d7\" (UID: \"86e3cacb-a113-4182-b955-fd45c0e77190\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" Oct 30 11:33:26.694630 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.694599 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/17a81880-2dca-4878-bf5a-6f7185f4451e-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2\" (UID: \"17a81880-2dca-4878-bf5a-6f7185f4451e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" Oct 30 11:33:26.694778 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.694756 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/17a81880-2dca-4878-bf5a-6f7185f4451e-config\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2\" (UID: \"17a81880-2dca-4878-bf5a-6f7185f4451e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" Oct 30 11:33:26.694832 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.694773 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/17a81880-2dca-4878-bf5a-6f7185f4451e-client-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2\" (UID: \"17a81880-2dca-4878-bf5a-6f7185f4451e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" Oct 30 11:33:26.694832 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.694760 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/86e3cacb-a113-4182-b955-fd45c0e77190-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8v5d7\" (UID: \"86e3cacb-a113-4182-b955-fd45c0e77190\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" Oct 30 11:33:26.696677 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.696654 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86e3cacb-a113-4182-b955-fd45c0e77190-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8v5d7\" (UID: \"86e3cacb-a113-4182-b955-fd45c0e77190\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" Oct 30 11:33:26.696755 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.696722 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/86e3cacb-a113-4182-b955-fd45c0e77190-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8v5d7\" (UID: \"86e3cacb-a113-4182-b955-fd45c0e77190\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" Oct 30 11:33:26.696822 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.696805 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/86e3cacb-a113-4182-b955-fd45c0e77190-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8v5d7\" (UID: \"86e3cacb-a113-4182-b955-fd45c0e77190\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" Oct 30 11:33:26.700369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.700344 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:26.717477 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.717398 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lv9g\" (UniqueName: \"kubernetes.io/projected/17a81880-2dca-4878-bf5a-6f7185f4451e-kube-api-access-6lv9g\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2\" (UID: \"17a81880-2dca-4878-bf5a-6f7185f4451e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" Oct 30 11:33:26.717550 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.717490 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqx2r\" (UniqueName: \"kubernetes.io/projected/86e3cacb-a113-4182-b955-fd45c0e77190-kube-api-access-lqx2r\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-8v5d7\" (UID: \"86e3cacb-a113-4182-b955-fd45c0e77190\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" Oct 30 11:33:26.773568 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.773535 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" Oct 30 11:33:26.795957 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.795925 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/098dce60-e115-4404-8e5f-43809c74e6b6-config\") pod \"korrel8r-5cbcc89b8f-xvlpg\" (UID: \"098dce60-e115-4404-8e5f-43809c74e6b6\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg" Oct 30 11:33:26.796110 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.795993 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/098dce60-e115-4404-8e5f-43809c74e6b6-tls\") pod \"korrel8r-5cbcc89b8f-xvlpg\" (UID: \"098dce60-e115-4404-8e5f-43809c74e6b6\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg" Oct 30 11:33:26.796110 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.796021 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2835390b-006c-4fc4-81fc-3ab4952f4dc6-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-hmd48\" (UID: \"2835390b-006c-4fc4-81fc-3ab4952f4dc6\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" Oct 30 11:33:26.796110 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.796051 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mprxd\" (UniqueName: \"kubernetes.io/projected/098dce60-e115-4404-8e5f-43809c74e6b6-kube-api-access-mprxd\") pod \"korrel8r-5cbcc89b8f-xvlpg\" (UID: \"098dce60-e115-4404-8e5f-43809c74e6b6\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg" Oct 30 11:33:26.796283 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.796124 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2835390b-006c-4fc4-81fc-3ab4952f4dc6-config\") pod \"data-science-thanos-querier-proxy-7b56895dd5-hmd48\" (UID: \"2835390b-006c-4fc4-81fc-3ab4952f4dc6\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" Oct 30 11:33:26.796283 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.796152 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpgwh\" (UniqueName: \"kubernetes.io/projected/2835390b-006c-4fc4-81fc-3ab4952f4dc6-kube-api-access-qpgwh\") pod \"data-science-thanos-querier-proxy-7b56895dd5-hmd48\" (UID: \"2835390b-006c-4fc4-81fc-3ab4952f4dc6\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" Oct 30 11:33:26.796764 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.796740 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2835390b-006c-4fc4-81fc-3ab4952f4dc6-config\") pod \"data-science-thanos-querier-proxy-7b56895dd5-hmd48\" (UID: \"2835390b-006c-4fc4-81fc-3ab4952f4dc6\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" Oct 30 11:33:26.796983 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.796952 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/098dce60-e115-4404-8e5f-43809c74e6b6-config\") pod \"korrel8r-5cbcc89b8f-xvlpg\" (UID: \"098dce60-e115-4404-8e5f-43809c74e6b6\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg" Oct 30 11:33:26.798615 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.798588 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2835390b-006c-4fc4-81fc-3ab4952f4dc6-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-hmd48\" (UID: \"2835390b-006c-4fc4-81fc-3ab4952f4dc6\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" Oct 30 11:33:26.798814 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.798797 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/098dce60-e115-4404-8e5f-43809c74e6b6-tls\") pod \"korrel8r-5cbcc89b8f-xvlpg\" (UID: \"098dce60-e115-4404-8e5f-43809c74e6b6\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg" Oct 30 11:33:26.819266 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.819237 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpgwh\" (UniqueName: \"kubernetes.io/projected/2835390b-006c-4fc4-81fc-3ab4952f4dc6-kube-api-access-qpgwh\") pod \"data-science-thanos-querier-proxy-7b56895dd5-hmd48\" (UID: \"2835390b-006c-4fc4-81fc-3ab4952f4dc6\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" Oct 30 11:33:26.819386 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.819331 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mprxd\" (UniqueName: \"kubernetes.io/projected/098dce60-e115-4404-8e5f-43809c74e6b6-kube-api-access-mprxd\") pod \"korrel8r-5cbcc89b8f-xvlpg\" (UID: \"098dce60-e115-4404-8e5f-43809c74e6b6\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg" Oct 30 11:33:26.828945 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.828914 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" Oct 30 11:33:26.861801 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.861768 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:26.872669 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:26.872635 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg" Oct 30 11:33:27.063479 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:27.063424 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:27.071285 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:27.071264 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7"] Oct 30 11:33:27.073627 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:33:27.073599 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e3cacb_a113_4182_b955_fd45c0e77190.slice/crio-3b054e557bd115e0401eb6963376e09b563b0e4c8dc7d7aa08bb3f531845d5be WatchSource:0}: Error finding container 3b054e557bd115e0401eb6963376e09b563b0e4c8dc7d7aa08bb3f531845d5be: Status 404 returned error can't find the container with id 3b054e557bd115e0401eb6963376e09b563b0e4c8dc7d7aa08bb3f531845d5be Oct 30 11:33:27.129174 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:27.129146 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48"] Oct 30 11:33:27.130940 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:33:27.130902 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2835390b_006c_4fc4_81fc_3ab4952f4dc6.slice/crio-78f214586f070210c68ed6c58fd7b0940bb7ca9012135d9e0ca57939dda52ecb WatchSource:0}: Error finding container 78f214586f070210c68ed6c58fd7b0940bb7ca9012135d9e0ca57939dda52ecb: Status 404 returned error can't find the container with id 78f214586f070210c68ed6c58fd7b0940bb7ca9012135d9e0ca57939dda52ecb Oct 30 11:33:27.174469 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:27.174444 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg"] Oct 30 11:33:27.176653 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:33:27.176629 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098dce60_e115_4404_8e5f_43809c74e6b6.slice/crio-b3db4a60bd9c6dd2c16c429db500ea52453160a6948bdcae2a64e559daae1827 WatchSource:0}: Error finding container b3db4a60bd9c6dd2c16c429db500ea52453160a6948bdcae2a64e559daae1827: Status 404 returned error can't find the container with id b3db4a60bd9c6dd2c16c429db500ea52453160a6948bdcae2a64e559daae1827 Oct 30 11:33:27.200662 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:27.200635 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/17a81880-2dca-4878-bf5a-6f7185f4451e-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2\" (UID: \"17a81880-2dca-4878-bf5a-6f7185f4451e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" Oct 30 11:33:27.202821 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:27.202800 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/17a81880-2dca-4878-bf5a-6f7185f4451e-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2\" (UID: \"17a81880-2dca-4878-bf5a-6f7185f4451e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" Oct 30 11:33:27.226677 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:27.226645 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:27.395847 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:27.395815 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" Oct 30 11:33:27.457604 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:27.457564 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:27.565840 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:27.565804 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" event={"ID":"2835390b-006c-4fc4-81fc-3ab4952f4dc6","Type":"ContainerStarted","Data":"ab6ddb4cf96c5fc5e978fe0fb7f028a7f0e8234d64601a77c0885e86e0a8a7cb"} Oct 30 11:33:27.566011 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:27.565877 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" event={"ID":"2835390b-006c-4fc4-81fc-3ab4952f4dc6","Type":"ContainerStarted","Data":"b98e3e5a6947742dce73f2a9df655696e64ea64e3905dc583d1213ce58eee593"} Oct 30 11:33:27.566011 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:27.565920 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" event={"ID":"2835390b-006c-4fc4-81fc-3ab4952f4dc6","Type":"ContainerStarted","Data":"f32449831bce2e1f1761659b07c517f700bd73ed9549d59cb1e84be9db5edb7c"} Oct 30 11:33:27.566011 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:27.565957 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" event={"ID":"2835390b-006c-4fc4-81fc-3ab4952f4dc6","Type":"ContainerStarted","Data":"78f214586f070210c68ed6c58fd7b0940bb7ca9012135d9e0ca57939dda52ecb"} Oct 30 11:33:27.567359 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:27.567317 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg" event={"ID":"098dce60-e115-4404-8e5f-43809c74e6b6","Type":"ContainerStarted","Data":"9684d7378a6e5ca8719fcfcfa5cd9b75a47821655099c926e72b62c8e043e3c6"} Oct 30 11:33:27.567490 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:27.567369 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg" event={"ID":"098dce60-e115-4404-8e5f-43809c74e6b6","Type":"ContainerStarted","Data":"b3db4a60bd9c6dd2c16c429db500ea52453160a6948bdcae2a64e559daae1827"} Oct 30 11:33:27.567490 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:27.567402 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg" Oct 30 11:33:27.568449 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:27.568406 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" event={"ID":"86e3cacb-a113-4182-b955-fd45c0e77190","Type":"ContainerStarted","Data":"3b054e557bd115e0401eb6963376e09b563b0e4c8dc7d7aa08bb3f531845d5be"} Oct 30 11:33:27.608651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:27.608605 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" podStartSLOduration=1.6085921349999999 podStartE2EDuration="1.608592135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:33:27.607113887 +0000 UTC m=+2898.108493372" watchObservedRunningTime="2026-10-30 11:33:27.608592135 +0000 UTC m=+2898.109971625" Oct 30 11:33:27.650712 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:27.650612 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg" podStartSLOduration=1.650597682 podStartE2EDuration="1.650597682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:33:27.649283312 +0000 UTC m=+2898.150662796" watchObservedRunningTime="2026-10-30 11:33:27.650597682 +0000 UTC m=+2898.151977167" Oct 30 11:33:27.693732 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:27.693708 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2"] Oct 30 11:33:27.695883 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:33:27.695846 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a81880_2dca_4878_bf5a_6f7185f4451e.slice/crio-aee5826d7f0f4cbae9064a68e26c82b9c9da6dcebe3486916dd5045cd2466c09 WatchSource:0}: Error finding container aee5826d7f0f4cbae9064a68e26c82b9c9da6dcebe3486916dd5045cd2466c09: Status 404 returned error can't find the container with id aee5826d7f0f4cbae9064a68e26c82b9c9da6dcebe3486916dd5045cd2466c09 Oct 30 11:33:28.015416 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:28.015381 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:28.186588 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:28.186511 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:28.574491 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:28.574451 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" event={"ID":"86e3cacb-a113-4182-b955-fd45c0e77190","Type":"ContainerStarted","Data":"b8c4a519d62c6f7b6f5be62b1c0f7d31ee31ea9bed6db61f9c8bf2531ee224c9"} Oct 30 11:33:28.574927 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:28.574604 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" Oct 30 11:33:28.576526 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:28.576505 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" event={"ID":"17a81880-2dca-4878-bf5a-6f7185f4451e","Type":"ContainerStarted","Data":"7e5b38c85078f1929b906b4de6d76e82b7fd55b8323d43ce5fc2c9bb7d4e509a"} Oct 30 11:33:28.576634 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:28.576559 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" event={"ID":"17a81880-2dca-4878-bf5a-6f7185f4451e","Type":"ContainerStarted","Data":"ee721e3921d2ae3ec4c2d6be3fe151ca597cb50375ba9aa82d914664aec59fec"} Oct 30 11:33:28.576634 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:28.576598 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" event={"ID":"17a81880-2dca-4878-bf5a-6f7185f4451e","Type":"ContainerStarted","Data":"cf4b519ce878052d3160b1237873c33573f01335ffb9abf6c71eea160bfcfdb4"} Oct 30 11:33:28.576634 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:28.576629 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" event={"ID":"17a81880-2dca-4878-bf5a-6f7185f4451e","Type":"ContainerStarted","Data":"aee5826d7f0f4cbae9064a68e26c82b9c9da6dcebe3486916dd5045cd2466c09"} Oct 30 11:33:28.615883 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:28.615833 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" podStartSLOduration=2.615820238 podStartE2EDuration="2.615820238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:33:28.615386518 +0000 UTC m=+2899.116766004" watchObservedRunningTime="2026-10-30 11:33:28.615820238 +0000 UTC m=+2899.117199723" Oct 30 11:33:28.657821 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:28.657773 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" podStartSLOduration=2.6577610529999998 podStartE2EDuration="2.657761053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:33:28.65691971 +0000 UTC m=+2899.158299195" watchObservedRunningTime="2026-10-30 11:33:28.657761053 +0000 UTC m=+2899.159140537" Oct 30 11:33:30.619690 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:30.619653 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:30.915976 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:30.915890 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:31.088042 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.088000 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:31.355981 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.355949 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 11:33:31.361097 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.361075 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:33:31.361234 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.361217 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.364967 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.364944 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:31.375066 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.375046 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 11:33:31.386370 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.386349 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 11:33:31.386487 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.386390 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 11:33:31.389133 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.388965 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 11:33:31.391747 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.391725 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:33:31.396019 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.395996 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:31.449085 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.449060 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 11:33:31.449258 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.449109 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 11:33:31.449258 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.449194 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 11:33:31.449369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.449280 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 11:33:31.449369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.449299 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 11:33:31.449369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.449342 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 11:33:31.449579 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.449489 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 11:33:31.449579 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.449536 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 11:33:31.449579 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.449534 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-c9rbj\"" Oct 30 11:33:31.449579 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.449551 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 11:33:31.449708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.449578 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-gg924\"" Oct 30 11:33:31.541025 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.540993 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.541025 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.541031 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.541257 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.541114 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fffff5fd-6828-41bf-88aa-d0e721fedbf0-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.541257 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.541148 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4c82f811-e9e7-4c22-ba2b-83e9419752ed-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:31.541257 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.541178 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fffff5fd-6828-41bf-88aa-d0e721fedbf0-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.541372 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.541281 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fffff5fd-6828-41bf-88aa-d0e721fedbf0-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.541372 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.541319 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.541474 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.541373 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.541474 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.541416 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c82f811-e9e7-4c22-ba2b-83e9419752ed-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:31.541474 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.541465 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4c82f811-e9e7-4c22-ba2b-83e9419752ed-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:31.541578 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.541483 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c82f811-e9e7-4c22-ba2b-83e9419752ed-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:31.541578 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.541504 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.541578 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.541519 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fffff5fd-6828-41bf-88aa-d0e721fedbf0-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.541578 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.541534 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fffff5fd-6828-41bf-88aa-d0e721fedbf0-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.541710 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.541603 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.541710 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.541633 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4c82f811-e9e7-4c22-ba2b-83e9419752ed-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:31.541710 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.541649 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8npm\" (UniqueName: \"kubernetes.io/projected/4c82f811-e9e7-4c22-ba2b-83e9419752ed-kube-api-access-b8npm\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:31.541710 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.541668 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-288n6\" (UniqueName: \"kubernetes.io/projected/fffff5fd-6828-41bf-88aa-d0e721fedbf0-kube-api-access-288n6\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.541710 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.541684 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c82f811-e9e7-4c22-ba2b-83e9419752ed-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:31.560316 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.560286 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:31.643121 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.643043 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.643121 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.643090 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c82f811-e9e7-4c22-ba2b-83e9419752ed-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:31.643121 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.643118 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4c82f811-e9e7-4c22-ba2b-83e9419752ed-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:31.643721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.643146 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c82f811-e9e7-4c22-ba2b-83e9419752ed-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:31.643721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.643178 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.643721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.643203 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fffff5fd-6828-41bf-88aa-d0e721fedbf0-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.643721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.643230 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fffff5fd-6828-41bf-88aa-d0e721fedbf0-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.643721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.643268 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.643721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.643291 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4c82f811-e9e7-4c22-ba2b-83e9419752ed-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:31.643721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.643314 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8npm\" (UniqueName: \"kubernetes.io/projected/4c82f811-e9e7-4c22-ba2b-83e9419752ed-kube-api-access-b8npm\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:31.643721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.643343 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-288n6\" (UniqueName: \"kubernetes.io/projected/fffff5fd-6828-41bf-88aa-d0e721fedbf0-kube-api-access-288n6\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.643721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.643369 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c82f811-e9e7-4c22-ba2b-83e9419752ed-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:31.643721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.643396 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.643721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.643425 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.643721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.643487 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fffff5fd-6828-41bf-88aa-d0e721fedbf0-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.643721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.643512 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4c82f811-e9e7-4c22-ba2b-83e9419752ed-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:31.643721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.643557 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fffff5fd-6828-41bf-88aa-d0e721fedbf0-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.643721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.643598 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fffff5fd-6828-41bf-88aa-d0e721fedbf0-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.643721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.643639 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.644568 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.643903 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fffff5fd-6828-41bf-88aa-d0e721fedbf0-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.644568 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.643958 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fffff5fd-6828-41bf-88aa-d0e721fedbf0-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.645889 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.644868 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4c82f811-e9e7-4c22-ba2b-83e9419752ed-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:31.646335 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.646306 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.646465 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.646414 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.646857 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.646829 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fffff5fd-6828-41bf-88aa-d0e721fedbf0-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.646990 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.646873 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c82f811-e9e7-4c22-ba2b-83e9419752ed-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:31.647549 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.647494 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.647729 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.647687 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.648453 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.648332 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4c82f811-e9e7-4c22-ba2b-83e9419752ed-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:31.648710 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.648685 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4c82f811-e9e7-4c22-ba2b-83e9419752ed-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:31.648968 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.648939 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.649092 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.649072 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c82f811-e9e7-4c22-ba2b-83e9419752ed-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:31.649332 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.649310 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.649394 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.649360 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fffff5fd-6828-41bf-88aa-d0e721fedbf0-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.649802 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.649783 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fffff5fd-6828-41bf-88aa-d0e721fedbf0-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.649878 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.649860 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c82f811-e9e7-4c22-ba2b-83e9419752ed-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:31.667616 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.667597 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-288n6\" (UniqueName: \"kubernetes.io/projected/fffff5fd-6828-41bf-88aa-d0e721fedbf0-kube-api-access-288n6\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:31.667762 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.667747 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8npm\" (UniqueName: \"kubernetes.io/projected/4c82f811-e9e7-4c22-ba2b-83e9419752ed-kube-api-access-b8npm\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:31.692719 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:31.692690 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:32.023511 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:32.023485 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:33:32.025587 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:33:32.025554 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c82f811_e9e7_4c22_ba2b_83e9419752ed.slice/crio-148bc5fe0e663903e41bd57957e5de2e8d19310462c1983dfe1d8dfb27c50bc4 WatchSource:0}: Error finding container 148bc5fe0e663903e41bd57957e5de2e8d19310462c1983dfe1d8dfb27c50bc4: Status 404 returned error can't find the container with id 148bc5fe0e663903e41bd57957e5de2e8d19310462c1983dfe1d8dfb27c50bc4 Oct 30 11:33:32.597400 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:32.597361 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"4c82f811-e9e7-4c22-ba2b-83e9419752ed","Type":"ContainerStarted","Data":"1d07d9edfe5f4982678252182b29f91cb2b57ebf3a51268711c8470db2811f53"} Oct 30 11:33:32.597620 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:32.597480 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"4c82f811-e9e7-4c22-ba2b-83e9419752ed","Type":"ContainerStarted","Data":"148bc5fe0e663903e41bd57957e5de2e8d19310462c1983dfe1d8dfb27c50bc4"} Oct 30 11:33:34.584623 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:34.584592 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" Oct 30 11:33:35.478621 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:35.478584 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ae2fc6cc-1a29-41df-b109-2b8489d4e098\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:35.508188 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:35.508163 4289 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-ae2fc6cc-1a29-41df-b109-2b8489d4e098\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:35.579499 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:35.579462 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-ae2fc6cc-1a29-41df-b109-2b8489d4e098\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:35.579637 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:35.579602 4289 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-ae2fc6cc-1a29-41df-b109-2b8489d4e098\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:35.594228 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:35.594202 4289 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-ae2fc6cc-1a29-41df-b109-2b8489d4e098\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") DevicePath \"csi-41fb3b285706a9985249f9bffb659d03160e9e3e6288d3728f95a76a8535f583\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:35.700706 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:35.700668 4289 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-ae2fc6cc-1a29-41df-b109-2b8489d4e098\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/5f4e377c3e259c071aab429e31ca61846b879cc3476e2f02b70e888ccdcdb590/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:35.727493 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:35.727465 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-ae2fc6cc-1a29-41df-b109-2b8489d4e098\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:35.884732 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:35.884632 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:36.215545 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:36.215519 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 11:33:36.216525 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:33:36.216496 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffff5fd_6828_41bf_88aa_d0e721fedbf0.slice/crio-402542ab3b822777f75520a0c231b0e250a091225f15d4eea92b7aaa52577dfa WatchSource:0}: Error finding container 402542ab3b822777f75520a0c231b0e250a091225f15d4eea92b7aaa52577dfa: Status 404 returned error can't find the container with id 402542ab3b822777f75520a0c231b0e250a091225f15d4eea92b7aaa52577dfa Oct 30 11:33:36.620485 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:36.620457 4289 generic.go:356] "Generic (PLEG): container finished" podID="4c82f811-e9e7-4c22-ba2b-83e9419752ed" containerID="1d07d9edfe5f4982678252182b29f91cb2b57ebf3a51268711c8470db2811f53" exitCode=0 Oct 30 11:33:36.620928 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:36.620530 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"4c82f811-e9e7-4c22-ba2b-83e9419752ed","Type":"ContainerDied","Data":"1d07d9edfe5f4982678252182b29f91cb2b57ebf3a51268711c8470db2811f53"} Oct 30 11:33:36.622138 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:36.622116 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fffff5fd-6828-41bf-88aa-d0e721fedbf0","Type":"ContainerStarted","Data":"7961334be6d6a4233aab05762344997aa429a124bbeaf121d6589c5a2f3e0df3"} Oct 30 11:33:36.622275 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:36.622177 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fffff5fd-6828-41bf-88aa-d0e721fedbf0","Type":"ContainerStarted","Data":"402542ab3b822777f75520a0c231b0e250a091225f15d4eea92b7aaa52577dfa"} Oct 30 11:33:37.627774 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:37.627694 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"4c82f811-e9e7-4c22-ba2b-83e9419752ed","Type":"ContainerStarted","Data":"fab688ee088d845b5a234e36b02d38d08d033d600d280016f6e5ac74b995905b"} Oct 30 11:33:37.627774 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:37.627755 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"4c82f811-e9e7-4c22-ba2b-83e9419752ed","Type":"ContainerStarted","Data":"81761ef6c91ecf83242e65ed237a1c6747f899843ace3fd6af253290b8d204c3"} Oct 30 11:33:37.628182 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:37.628061 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:37.702515 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:37.702467 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.417907854 podStartE2EDuration="6.70245393s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.284546076s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:33:37.689051809 +0000 UTC m=+2908.190431296" watchObservedRunningTime="2026-10-30 11:33:37.70245393 +0000 UTC m=+2908.203833411" Oct 30 11:33:38.578848 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:38.578818 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg" Oct 30 11:33:40.643999 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:40.643964 4289 generic.go:356] "Generic (PLEG): container finished" podID="fffff5fd-6828-41bf-88aa-d0e721fedbf0" containerID="7961334be6d6a4233aab05762344997aa429a124bbeaf121d6589c5a2f3e0df3" exitCode=0 Oct 30 11:33:40.644372 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:40.644035 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fffff5fd-6828-41bf-88aa-d0e721fedbf0","Type":"ContainerDied","Data":"7961334be6d6a4233aab05762344997aa429a124bbeaf121d6589c5a2f3e0df3"} Oct 30 11:33:41.650616 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:41.650529 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fffff5fd-6828-41bf-88aa-d0e721fedbf0","Type":"ContainerStarted","Data":"65195ead31a8723aa11b20f1e70b1604acb5d2af76798261fc58d4a5f39cfc5e"} Oct 30 11:33:41.650616 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:41.650595 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fffff5fd-6828-41bf-88aa-d0e721fedbf0","Type":"ContainerStarted","Data":"fafd07d8121393996f64ebec92eeb3b54c2ffe2f52913f42bc9e058b3a5047e6"} Oct 30 11:33:41.651081 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:41.650619 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fffff5fd-6828-41bf-88aa-d0e721fedbf0","Type":"ContainerStarted","Data":"b0e5c885df440ef973a805a54acb26e3335937f8893c81ce3589e2afe49b7232"} Oct 30 11:33:41.728760 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:41.728695 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=7.342680111 podStartE2EDuration="10.728677643s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.385997532s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:33:41.716217439 +0000 UTC m=+2912.217596924" watchObservedRunningTime="2026-10-30 11:33:41.728677643 +0000 UTC m=+2912.230057129" Oct 30 11:33:43.635010 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:43.634983 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:33:45.238960 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:45.238926 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:45.439156 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:45.439122 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:45.639480 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:45.639383 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:45.885390 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:45.885356 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:47.040307 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.040273 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:47.221282 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.221246 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:47.429024 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.428950 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:33:47.432652 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.432631 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:47.460076 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.460050 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 11:33:47.460207 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.460160 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 11:33:47.460267 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.460241 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 11:33:47.460313 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.460285 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-e4cadd38\"" Oct 30 11:33:47.474281 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.474254 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:33:47.548353 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.548322 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dr7t6\"" Oct 30 11:33:47.595014 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.594976 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6e502a3e-98cf-472e-8259-339107e5f156-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"6e502a3e-98cf-472e-8259-339107e5f156\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:47.595227 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.595031 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e502a3e-98cf-472e-8259-339107e5f156-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6e502a3e-98cf-472e-8259-339107e5f156\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:47.595227 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.595107 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5r42\" (UniqueName: \"kubernetes.io/projected/6e502a3e-98cf-472e-8259-339107e5f156-kube-api-access-g5r42\") pod \"data-science-collector-collector-1\" (UID: \"6e502a3e-98cf-472e-8259-339107e5f156\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:47.595227 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.595198 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e502a3e-98cf-472e-8259-339107e5f156-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6e502a3e-98cf-472e-8259-339107e5f156\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:47.595388 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.595304 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6e502a3e-98cf-472e-8259-339107e5f156-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"6e502a3e-98cf-472e-8259-339107e5f156\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:47.696687 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.696596 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e502a3e-98cf-472e-8259-339107e5f156-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6e502a3e-98cf-472e-8259-339107e5f156\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:47.696687 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.696634 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5r42\" (UniqueName: \"kubernetes.io/projected/6e502a3e-98cf-472e-8259-339107e5f156-kube-api-access-g5r42\") pod \"data-science-collector-collector-1\" (UID: \"6e502a3e-98cf-472e-8259-339107e5f156\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:47.696687 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.696682 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e502a3e-98cf-472e-8259-339107e5f156-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6e502a3e-98cf-472e-8259-339107e5f156\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:47.697173 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.696749 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6e502a3e-98cf-472e-8259-339107e5f156-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"6e502a3e-98cf-472e-8259-339107e5f156\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:47.697173 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.696787 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6e502a3e-98cf-472e-8259-339107e5f156-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"6e502a3e-98cf-472e-8259-339107e5f156\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:47.697762 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.697738 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6e502a3e-98cf-472e-8259-339107e5f156-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"6e502a3e-98cf-472e-8259-339107e5f156\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:47.699483 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.699456 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e502a3e-98cf-472e-8259-339107e5f156-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6e502a3e-98cf-472e-8259-339107e5f156\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:47.699768 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.699747 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6e502a3e-98cf-472e-8259-339107e5f156-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"6e502a3e-98cf-472e-8259-339107e5f156\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:47.699862 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.699750 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e502a3e-98cf-472e-8259-339107e5f156-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6e502a3e-98cf-472e-8259-339107e5f156\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:47.723177 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.723154 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5r42\" (UniqueName: \"kubernetes.io/projected/6e502a3e-98cf-472e-8259-339107e5f156-kube-api-access-g5r42\") pod \"data-science-collector-collector-1\" (UID: \"6e502a3e-98cf-472e-8259-339107e5f156\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:47.745167 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:47.745141 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:48.045011 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:48.044985 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:33:48.047122 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:33:48.047094 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e502a3e_98cf_472e_8259_339107e5f156.slice/crio-bace935c1f65d732fe8752b69ed5d669a15783ce2b45e5f135d97412ab776626 WatchSource:0}: Error finding container bace935c1f65d732fe8752b69ed5d669a15783ce2b45e5f135d97412ab776626: Status 404 returned error can't find the container with id bace935c1f65d732fe8752b69ed5d669a15783ce2b45e5f135d97412ab776626 Oct 30 11:33:48.049365 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:48.049333 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:33:48.049517 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:48.049397 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:33:48.693924 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:48.693880 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6e502a3e-98cf-472e-8259-339107e5f156","Type":"ContainerStarted","Data":"250a3b6bfcadaf79121033e7b2eb64692ca8e241b4638cdef57c368f30112385"} Oct 30 11:33:48.694115 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:48.693970 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6e502a3e-98cf-472e-8259-339107e5f156","Type":"ContainerStarted","Data":"bace935c1f65d732fe8752b69ed5d669a15783ce2b45e5f135d97412ab776626"} Oct 30 11:33:48.737501 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:48.737443 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.7374154800000001 podStartE2EDuration="1.73741548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:33:48.735593874 +0000 UTC m=+2919.236973371" watchObservedRunningTime="2026-10-30 11:33:48.73741548 +0000 UTC m=+2919.238794964" Oct 30 11:33:50.885597 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:50.885567 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:50.891078 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:50.891056 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:51.712406 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:51.712376 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:33:55.614107 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:55.613997 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:55.811451 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:55.811391 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:55.974572 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:55.974545 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:56.257752 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:56.257668 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:33:57.240195 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.240159 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:33:57.240653 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.240361 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="6e502a3e-98cf-472e-8259-339107e5f156" containerName="otc-container" containerID="cri-o://250a3b6bfcadaf79121033e7b2eb64692ca8e241b4638cdef57c368f30112385" gracePeriod=30 Oct 30 11:33:57.421259 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.421209 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:57.484193 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.484165 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:57.489639 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.489618 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e502a3e-98cf-472e-8259-339107e5f156-kube-api-access-g5r42\" (UniqueName: \"kubernetes.io/projected/6e502a3e-98cf-472e-8259-339107e5f156-kube-api-access-g5r42\") pod \"6e502a3e-98cf-472e-8259-339107e5f156\" (UID: \"6e502a3e-98cf-472e-8259-339107e5f156\") " Oct 30 11:33:57.489729 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.489650 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e502a3e-98cf-472e-8259-339107e5f156-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e502a3e-98cf-472e-8259-339107e5f156-tls-certs\") pod \"6e502a3e-98cf-472e-8259-339107e5f156\" (UID: \"6e502a3e-98cf-472e-8259-339107e5f156\") " Oct 30 11:33:57.489729 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.489692 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e502a3e-98cf-472e-8259-339107e5f156-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e502a3e-98cf-472e-8259-339107e5f156-monitoring-tls-certs\") pod \"6e502a3e-98cf-472e-8259-339107e5f156\" (UID: \"6e502a3e-98cf-472e-8259-339107e5f156\") " Oct 30 11:33:57.489835 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.489741 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e502a3e-98cf-472e-8259-339107e5f156-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6e502a3e-98cf-472e-8259-339107e5f156-data-science-collector-ta-client-cert\") pod \"6e502a3e-98cf-472e-8259-339107e5f156\" (UID: \"6e502a3e-98cf-472e-8259-339107e5f156\") " Oct 30 11:33:57.489835 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.489768 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e502a3e-98cf-472e-8259-339107e5f156-otc-internal\" (UniqueName: \"kubernetes.io/configmap/6e502a3e-98cf-472e-8259-339107e5f156-otc-internal\") pod \"6e502a3e-98cf-472e-8259-339107e5f156\" (UID: \"6e502a3e-98cf-472e-8259-339107e5f156\") " Oct 30 11:33:57.490256 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.490185 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e502a3e-98cf-472e-8259-339107e5f156-otc-internal" pod "6e502a3e-98cf-472e-8259-339107e5f156" (UID: "6e502a3e-98cf-472e-8259-339107e5f156"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:33:57.491842 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.491801 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e502a3e-98cf-472e-8259-339107e5f156-tls-certs" pod "6e502a3e-98cf-472e-8259-339107e5f156" (UID: "6e502a3e-98cf-472e-8259-339107e5f156"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:33:57.491962 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.491909 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e502a3e-98cf-472e-8259-339107e5f156-kube-api-access-g5r42" pod "6e502a3e-98cf-472e-8259-339107e5f156" (UID: "6e502a3e-98cf-472e-8259-339107e5f156"). InnerVolumeSpecName "kube-api-access-g5r42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:33:57.492185 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.492168 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e502a3e-98cf-472e-8259-339107e5f156-monitoring-tls-certs" pod "6e502a3e-98cf-472e-8259-339107e5f156" (UID: "6e502a3e-98cf-472e-8259-339107e5f156"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:33:57.492247 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.492196 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e502a3e-98cf-472e-8259-339107e5f156-data-science-collector-ta-client-cert" pod "6e502a3e-98cf-472e-8259-339107e5f156" (UID: "6e502a3e-98cf-472e-8259-339107e5f156"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:33:57.590499 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.590459 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6e502a3e-98cf-472e-8259-339107e5f156-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:57.590499 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.590488 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6e502a3e-98cf-472e-8259-339107e5f156-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:57.590499 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.590498 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5r42\" (UniqueName: \"kubernetes.io/projected/6e502a3e-98cf-472e-8259-339107e5f156-kube-api-access-g5r42\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:57.590499 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.590507 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e502a3e-98cf-472e-8259-339107e5f156-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:57.590905 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.590515 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e502a3e-98cf-472e-8259-339107e5f156-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:33:57.590905 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.590825 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:57.736868 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.736833 4289 generic.go:356] "Generic (PLEG): container finished" podID="6e502a3e-98cf-472e-8259-339107e5f156" containerID="250a3b6bfcadaf79121033e7b2eb64692ca8e241b4638cdef57c368f30112385" exitCode=0 Oct 30 11:33:57.737108 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.736887 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:57.737108 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.736885 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6e502a3e-98cf-472e-8259-339107e5f156","Type":"ContainerDied","Data":"250a3b6bfcadaf79121033e7b2eb64692ca8e241b4638cdef57c368f30112385"} Oct 30 11:33:57.737108 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.737028 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6e502a3e-98cf-472e-8259-339107e5f156","Type":"ContainerDied","Data":"bace935c1f65d732fe8752b69ed5d669a15783ce2b45e5f135d97412ab776626"} Oct 30 11:33:57.737108 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.737069 4289 scope.go:122] "RemoveContainer" containerID="250a3b6bfcadaf79121033e7b2eb64692ca8e241b4638cdef57c368f30112385" Oct 30 11:33:57.745721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.745701 4289 scope.go:122] "RemoveContainer" containerID="250a3b6bfcadaf79121033e7b2eb64692ca8e241b4638cdef57c368f30112385" Oct 30 11:33:57.746001 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:33:57.745980 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"250a3b6bfcadaf79121033e7b2eb64692ca8e241b4638cdef57c368f30112385\": container with ID starting with 250a3b6bfcadaf79121033e7b2eb64692ca8e241b4638cdef57c368f30112385 not found: ID does not exist" containerID="250a3b6bfcadaf79121033e7b2eb64692ca8e241b4638cdef57c368f30112385" Oct 30 11:33:57.746040 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.746014 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"250a3b6bfcadaf79121033e7b2eb64692ca8e241b4638cdef57c368f30112385"} err="failed to get container status \"250a3b6bfcadaf79121033e7b2eb64692ca8e241b4638cdef57c368f30112385\": rpc error: code = NotFound desc = could not find container \"250a3b6bfcadaf79121033e7b2eb64692ca8e241b4638cdef57c368f30112385\": container with ID starting with 250a3b6bfcadaf79121033e7b2eb64692ca8e241b4638cdef57c368f30112385 not found: ID does not exist" Oct 30 11:33:57.788386 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.788356 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:33:57.792583 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.792550 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:33:57.830793 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.830761 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:33:57.831173 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.831160 4289 cpu_manager.go:414] "containerMap: removing container" podUID="6e502a3e-98cf-472e-8259-339107e5f156" containerName="otc-container" Oct 30 11:33:57.831220 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.831173 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e502a3e-98cf-472e-8259-339107e5f156" containerName="otc-container" Oct 30 11:33:57.831258 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.831242 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e502a3e-98cf-472e-8259-339107e5f156" containerName="otc-container" Oct 30 11:33:57.835686 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.835670 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:57.854646 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.854621 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 11:33:57.855097 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.854689 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-e4cadd38\"" Oct 30 11:33:57.855097 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.854745 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dr7t6\"" Oct 30 11:33:57.855097 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.854795 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 11:33:57.855097 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.854822 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 11:33:57.855575 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.855557 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:33:57.893540 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.893497 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsxck\" (UniqueName: \"kubernetes.io/projected/50391e47-44b9-4939-8eec-0e53758e5cae-kube-api-access-wsxck\") pod \"data-science-collector-collector-1\" (UID: \"50391e47-44b9-4939-8eec-0e53758e5cae\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:57.893729 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.893640 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/50391e47-44b9-4939-8eec-0e53758e5cae-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"50391e47-44b9-4939-8eec-0e53758e5cae\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:57.893729 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.893670 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/50391e47-44b9-4939-8eec-0e53758e5cae-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"50391e47-44b9-4939-8eec-0e53758e5cae\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:57.893729 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.893697 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/50391e47-44b9-4939-8eec-0e53758e5cae-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"50391e47-44b9-4939-8eec-0e53758e5cae\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:57.893909 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.893829 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50391e47-44b9-4939-8eec-0e53758e5cae-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"50391e47-44b9-4939-8eec-0e53758e5cae\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:57.994595 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.994554 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/50391e47-44b9-4939-8eec-0e53758e5cae-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"50391e47-44b9-4939-8eec-0e53758e5cae\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:57.994595 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.994595 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/50391e47-44b9-4939-8eec-0e53758e5cae-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"50391e47-44b9-4939-8eec-0e53758e5cae\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:57.994823 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.994626 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/50391e47-44b9-4939-8eec-0e53758e5cae-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"50391e47-44b9-4939-8eec-0e53758e5cae\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:57.994823 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.994692 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50391e47-44b9-4939-8eec-0e53758e5cae-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"50391e47-44b9-4939-8eec-0e53758e5cae\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:57.994823 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.994721 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsxck\" (UniqueName: \"kubernetes.io/projected/50391e47-44b9-4939-8eec-0e53758e5cae-kube-api-access-wsxck\") pod \"data-science-collector-collector-1\" (UID: \"50391e47-44b9-4939-8eec-0e53758e5cae\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:57.995366 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.995340 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/50391e47-44b9-4939-8eec-0e53758e5cae-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"50391e47-44b9-4939-8eec-0e53758e5cae\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:57.997199 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.997153 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50391e47-44b9-4939-8eec-0e53758e5cae-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"50391e47-44b9-4939-8eec-0e53758e5cae\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:57.997355 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.997332 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/50391e47-44b9-4939-8eec-0e53758e5cae-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"50391e47-44b9-4939-8eec-0e53758e5cae\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:57.997396 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:57.997355 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/50391e47-44b9-4939-8eec-0e53758e5cae-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"50391e47-44b9-4939-8eec-0e53758e5cae\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:58.017874 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:58.017849 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsxck\" (UniqueName: \"kubernetes.io/projected/50391e47-44b9-4939-8eec-0e53758e5cae-kube-api-access-wsxck\") pod \"data-science-collector-collector-1\" (UID: \"50391e47-44b9-4939-8eec-0e53758e5cae\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:58.148095 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:58.148060 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:33:58.447971 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:58.447946 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:33:58.450062 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:33:58.450035 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50391e47_44b9_4939_8eec_0e53758e5cae.slice/crio-ed3ef828a2ec71ea529ae530350ca1a0c6cc4cf0db6338f38769c39061160378 WatchSource:0}: Error finding container ed3ef828a2ec71ea529ae530350ca1a0c6cc4cf0db6338f38769c39061160378: Status 404 returned error can't find the container with id ed3ef828a2ec71ea529ae530350ca1a0c6cc4cf0db6338f38769c39061160378 Oct 30 11:33:58.452928 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:58.452893 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:33:58.453031 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:58.452956 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:33:58.743154 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:58.743116 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"50391e47-44b9-4939-8eec-0e53758e5cae","Type":"ContainerStarted","Data":"233ecd11ed973e022ae45e132ee460a7ca0a40cae8ac705fab1220d567e5fc87"} Oct 30 11:33:58.743339 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:58.743177 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"50391e47-44b9-4939-8eec-0e53758e5cae","Type":"ContainerStarted","Data":"ed3ef828a2ec71ea529ae530350ca1a0c6cc4cf0db6338f38769c39061160378"} Oct 30 11:33:58.786313 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:58.786252 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.786237867 podStartE2EDuration="1.786237867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:33:58.784910967 +0000 UTC m=+2929.286290453" watchObservedRunningTime="2026-10-30 11:33:58.786237867 +0000 UTC m=+2929.287617409" Oct 30 11:33:59.109664 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:59.109584 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:33:59.624376 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:33:59.624331 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e502a3e-98cf-472e-8259-339107e5f156" path="/var/lib/kubelet/pods/6e502a3e-98cf-472e-8259-339107e5f156/volumes" Oct 30 11:34:01.043032 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.042997 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7"] Oct 30 11:34:01.043463 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.043197 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" podUID="86e3cacb-a113-4182-b955-fd45c0e77190" containerName="kube-rbac-proxy" containerID="cri-o://b8c4a519d62c6f7b6f5be62b1c0f7d31ee31ea9bed6db61f9c8bf2531ee224c9" gracePeriod=30 Oct 30 11:34:01.056353 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.056323 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2"] Oct 30 11:34:01.056739 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.056713 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" podUID="17a81880-2dca-4878-bf5a-6f7185f4451e" containerName="method-gate" containerID="cri-o://ee721e3921d2ae3ec4c2d6be3fe151ca597cb50375ba9aa82d914664aec59fec" gracePeriod=30 Oct 30 11:34:01.056808 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.056710 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" podUID="17a81880-2dca-4878-bf5a-6f7185f4451e" containerName="kube-rbac-proxy" containerID="cri-o://cf4b519ce878052d3160b1237873c33573f01335ffb9abf6c71eea160bfcfdb4" gracePeriod=30 Oct 30 11:34:01.056869 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.056714 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" podUID="17a81880-2dca-4878-bf5a-6f7185f4451e" containerName="prom-label-proxy" containerID="cri-o://7e5b38c85078f1929b906b4de6d76e82b7fd55b8323d43ce5fc2c9bb7d4e509a" gracePeriod=30 Oct 30 11:34:01.076126 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.076099 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48"] Oct 30 11:34:01.076329 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.076308 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" podUID="2835390b-006c-4fc4-81fc-3ab4952f4dc6" containerName="kube-rbac-proxy" containerID="cri-o://f32449831bce2e1f1761659b07c517f700bd73ed9549d59cb1e84be9db5edb7c" gracePeriod=30 Oct 30 11:34:01.076329 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.076317 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" podUID="2835390b-006c-4fc4-81fc-3ab4952f4dc6" containerName="prom-label-proxy" containerID="cri-o://ab6ddb4cf96c5fc5e978fe0fb7f028a7f0e8234d64601a77c0885e86e0a8a7cb" gracePeriod=30 Oct 30 11:34:01.076490 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.076345 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" podUID="2835390b-006c-4fc4-81fc-3ab4952f4dc6" containerName="method-gate" containerID="cri-o://b98e3e5a6947742dce73f2a9df655696e64ea64e3905dc583d1213ce58eee593" gracePeriod=30 Oct 30 11:34:01.091748 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.091722 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg"] Oct 30 11:34:01.091930 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.091911 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg" podUID="098dce60-e115-4404-8e5f-43809c74e6b6" containerName="korrel8r" containerID="cri-o://9684d7378a6e5ca8719fcfcfa5cd9b75a47821655099c926e72b62c8e043e3c6" gracePeriod=30 Oct 30 11:34:01.350013 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.349985 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg" Oct 30 11:34:01.360645 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.360620 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" Oct 30 11:34:01.413676 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.413629 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" Oct 30 11:34:01.423707 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.423676 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" Oct 30 11:34:01.432415 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.432386 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a81880-2dca-4878-bf5a-6f7185f4451e-kube-api-access-6lv9g\" (UniqueName: \"kubernetes.io/projected/17a81880-2dca-4878-bf5a-6f7185f4451e-kube-api-access-6lv9g\") pod \"17a81880-2dca-4878-bf5a-6f7185f4451e\" (UID: \"17a81880-2dca-4878-bf5a-6f7185f4451e\") " Oct 30 11:34:01.432571 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.432479 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17a81880-2dca-4878-bf5a-6f7185f4451e-client-ca\" (UniqueName: \"kubernetes.io/configmap/17a81880-2dca-4878-bf5a-6f7185f4451e-client-ca\") pod \"17a81880-2dca-4878-bf5a-6f7185f4451e\" (UID: \"17a81880-2dca-4878-bf5a-6f7185f4451e\") " Oct 30 11:34:01.432571 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.432515 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/098dce60-e115-4404-8e5f-43809c74e6b6-kube-api-access-mprxd\" (UniqueName: \"kubernetes.io/projected/098dce60-e115-4404-8e5f-43809c74e6b6-kube-api-access-mprxd\") pod \"098dce60-e115-4404-8e5f-43809c74e6b6\" (UID: \"098dce60-e115-4404-8e5f-43809c74e6b6\") " Oct 30 11:34:01.432571 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.432553 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86e3cacb-a113-4182-b955-fd45c0e77190-kube-api-access-lqx2r\" (UniqueName: \"kubernetes.io/projected/86e3cacb-a113-4182-b955-fd45c0e77190-kube-api-access-lqx2r\") pod \"86e3cacb-a113-4182-b955-fd45c0e77190\" (UID: \"86e3cacb-a113-4182-b955-fd45c0e77190\") " Oct 30 11:34:01.432946 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.432836 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a81880-2dca-4878-bf5a-6f7185f4451e-client-ca" pod "17a81880-2dca-4878-bf5a-6f7185f4451e" (UID: "17a81880-2dca-4878-bf5a-6f7185f4451e"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:34:01.433278 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.433046 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/098dce60-e115-4404-8e5f-43809c74e6b6-config\" (UniqueName: \"kubernetes.io/configmap/098dce60-e115-4404-8e5f-43809c74e6b6-config\") pod \"098dce60-e115-4404-8e5f-43809c74e6b6\" (UID: \"098dce60-e115-4404-8e5f-43809c74e6b6\") " Oct 30 11:34:01.434008 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.433985 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/098dce60-e115-4404-8e5f-43809c74e6b6-config" pod "098dce60-e115-4404-8e5f-43809c74e6b6" (UID: "098dce60-e115-4404-8e5f-43809c74e6b6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:34:01.434403 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.434379 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/098dce60-e115-4404-8e5f-43809c74e6b6-tls\" (UniqueName: \"kubernetes.io/secret/098dce60-e115-4404-8e5f-43809c74e6b6-tls\") pod \"098dce60-e115-4404-8e5f-43809c74e6b6\" (UID: \"098dce60-e115-4404-8e5f-43809c74e6b6\") " Oct 30 11:34:01.434521 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.434456 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86e3cacb-a113-4182-b955-fd45c0e77190-tls\" (UniqueName: \"kubernetes.io/secret/86e3cacb-a113-4182-b955-fd45c0e77190-tls\") pod \"86e3cacb-a113-4182-b955-fd45c0e77190\" (UID: \"86e3cacb-a113-4182-b955-fd45c0e77190\") " Oct 30 11:34:01.434521 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.434503 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17a81880-2dca-4878-bf5a-6f7185f4451e-tls\" (UniqueName: \"kubernetes.io/secret/17a81880-2dca-4878-bf5a-6f7185f4451e-tls\") pod \"17a81880-2dca-4878-bf5a-6f7185f4451e\" (UID: \"17a81880-2dca-4878-bf5a-6f7185f4451e\") " Oct 30 11:34:01.436772 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.434557 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17a81880-2dca-4878-bf5a-6f7185f4451e-config\" (UniqueName: \"kubernetes.io/configmap/17a81880-2dca-4878-bf5a-6f7185f4451e-config\") pod \"17a81880-2dca-4878-bf5a-6f7185f4451e\" (UID: \"17a81880-2dca-4878-bf5a-6f7185f4451e\") " Oct 30 11:34:01.436772 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.434626 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17a81880-2dca-4878-bf5a-6f7185f4451e-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/17a81880-2dca-4878-bf5a-6f7185f4451e-prometheus-tls-ca\") pod \"17a81880-2dca-4878-bf5a-6f7185f4451e\" (UID: \"17a81880-2dca-4878-bf5a-6f7185f4451e\") " Oct 30 11:34:01.436772 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.434733 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86e3cacb-a113-4182-b955-fd45c0e77190-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/86e3cacb-a113-4182-b955-fd45c0e77190-prometheus-client-cert\") pod \"86e3cacb-a113-4182-b955-fd45c0e77190\" (UID: \"86e3cacb-a113-4182-b955-fd45c0e77190\") " Oct 30 11:34:01.436772 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.434824 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86e3cacb-a113-4182-b955-fd45c0e77190-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/86e3cacb-a113-4182-b955-fd45c0e77190-prometheus-ca\") pod \"86e3cacb-a113-4182-b955-fd45c0e77190\" (UID: \"86e3cacb-a113-4182-b955-fd45c0e77190\") " Oct 30 11:34:01.436772 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.434881 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86e3cacb-a113-4182-b955-fd45c0e77190-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86e3cacb-a113-4182-b955-fd45c0e77190-kube-rbac-proxy-config\") pod \"86e3cacb-a113-4182-b955-fd45c0e77190\" (UID: \"86e3cacb-a113-4182-b955-fd45c0e77190\") " Oct 30 11:34:01.436772 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.435222 4289 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/17a81880-2dca-4878-bf5a-6f7185f4451e-client-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:01.436772 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.435241 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/098dce60-e115-4404-8e5f-43809c74e6b6-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:01.436772 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.435720 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a81880-2dca-4878-bf5a-6f7185f4451e-config" pod "17a81880-2dca-4878-bf5a-6f7185f4451e" (UID: "17a81880-2dca-4878-bf5a-6f7185f4451e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:34:01.436772 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.436274 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a81880-2dca-4878-bf5a-6f7185f4451e-prometheus-tls-ca" pod "17a81880-2dca-4878-bf5a-6f7185f4451e" (UID: "17a81880-2dca-4878-bf5a-6f7185f4451e"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:34:01.436772 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.436689 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86e3cacb-a113-4182-b955-fd45c0e77190-prometheus-ca" pod "86e3cacb-a113-4182-b955-fd45c0e77190" (UID: "86e3cacb-a113-4182-b955-fd45c0e77190"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:34:01.437786 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.437578 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e3cacb-a113-4182-b955-fd45c0e77190-kube-api-access-lqx2r" pod "86e3cacb-a113-4182-b955-fd45c0e77190" (UID: "86e3cacb-a113-4182-b955-fd45c0e77190"). InnerVolumeSpecName "kube-api-access-lqx2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:34:01.437786 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.437739 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a81880-2dca-4878-bf5a-6f7185f4451e-kube-api-access-6lv9g" pod "17a81880-2dca-4878-bf5a-6f7185f4451e" (UID: "17a81880-2dca-4878-bf5a-6f7185f4451e"). InnerVolumeSpecName "kube-api-access-6lv9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:34:01.438671 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.438646 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098dce60-e115-4404-8e5f-43809c74e6b6-kube-api-access-mprxd" pod "098dce60-e115-4404-8e5f-43809c74e6b6" (UID: "098dce60-e115-4404-8e5f-43809c74e6b6"). InnerVolumeSpecName "kube-api-access-mprxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:34:01.439895 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.439848 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86e3cacb-a113-4182-b955-fd45c0e77190-prometheus-client-cert" pod "86e3cacb-a113-4182-b955-fd45c0e77190" (UID: "86e3cacb-a113-4182-b955-fd45c0e77190"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:01.439895 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.439872 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a81880-2dca-4878-bf5a-6f7185f4451e-tls" pod "17a81880-2dca-4878-bf5a-6f7185f4451e" (UID: "17a81880-2dca-4878-bf5a-6f7185f4451e"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:01.440045 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.440027 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86e3cacb-a113-4182-b955-fd45c0e77190-tls" pod "86e3cacb-a113-4182-b955-fd45c0e77190" (UID: "86e3cacb-a113-4182-b955-fd45c0e77190"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:01.441483 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.441212 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86e3cacb-a113-4182-b955-fd45c0e77190-kube-rbac-proxy-config" pod "86e3cacb-a113-4182-b955-fd45c0e77190" (UID: "86e3cacb-a113-4182-b955-fd45c0e77190"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:01.448607 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.448580 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/098dce60-e115-4404-8e5f-43809c74e6b6-tls" pod "098dce60-e115-4404-8e5f-43809c74e6b6" (UID: "098dce60-e115-4404-8e5f-43809c74e6b6"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:01.504232 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.504193 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:01.537708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.537673 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2835390b-006c-4fc4-81fc-3ab4952f4dc6-kube-api-access-qpgwh\" (UniqueName: \"kubernetes.io/projected/2835390b-006c-4fc4-81fc-3ab4952f4dc6-kube-api-access-qpgwh\") pod \"2835390b-006c-4fc4-81fc-3ab4952f4dc6\" (UID: \"2835390b-006c-4fc4-81fc-3ab4952f4dc6\") " Oct 30 11:34:01.537877 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.537743 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2835390b-006c-4fc4-81fc-3ab4952f4dc6-tls\" (UniqueName: \"kubernetes.io/secret/2835390b-006c-4fc4-81fc-3ab4952f4dc6-tls\") pod \"2835390b-006c-4fc4-81fc-3ab4952f4dc6\" (UID: \"2835390b-006c-4fc4-81fc-3ab4952f4dc6\") " Oct 30 11:34:01.537877 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.537812 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2835390b-006c-4fc4-81fc-3ab4952f4dc6-config\" (UniqueName: \"kubernetes.io/configmap/2835390b-006c-4fc4-81fc-3ab4952f4dc6-config\") pod \"2835390b-006c-4fc4-81fc-3ab4952f4dc6\" (UID: \"2835390b-006c-4fc4-81fc-3ab4952f4dc6\") " Oct 30 11:34:01.538170 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.538143 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/098dce60-e115-4404-8e5f-43809c74e6b6-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:01.538170 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.538171 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/86e3cacb-a113-4182-b955-fd45c0e77190-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:01.538351 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.538183 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/17a81880-2dca-4878-bf5a-6f7185f4451e-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:01.538351 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.538196 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/17a81880-2dca-4878-bf5a-6f7185f4451e-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:01.538351 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.538209 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/17a81880-2dca-4878-bf5a-6f7185f4451e-prometheus-tls-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:01.538351 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.538224 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/86e3cacb-a113-4182-b955-fd45c0e77190-prometheus-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:01.538351 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.538239 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/86e3cacb-a113-4182-b955-fd45c0e77190-prometheus-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:01.538351 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.538254 4289 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86e3cacb-a113-4182-b955-fd45c0e77190-kube-rbac-proxy-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:01.538351 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.538269 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lv9g\" (UniqueName: \"kubernetes.io/projected/17a81880-2dca-4878-bf5a-6f7185f4451e-kube-api-access-6lv9g\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:01.538351 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.538283 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mprxd\" (UniqueName: \"kubernetes.io/projected/098dce60-e115-4404-8e5f-43809c74e6b6-kube-api-access-mprxd\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:01.538351 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.538296 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqx2r\" (UniqueName: \"kubernetes.io/projected/86e3cacb-a113-4182-b955-fd45c0e77190-kube-api-access-lqx2r\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:01.543377 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.543344 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2835390b-006c-4fc4-81fc-3ab4952f4dc6-config" pod "2835390b-006c-4fc4-81fc-3ab4952f4dc6" (UID: "2835390b-006c-4fc4-81fc-3ab4952f4dc6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:34:01.544232 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.544184 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2835390b-006c-4fc4-81fc-3ab4952f4dc6-tls" pod "2835390b-006c-4fc4-81fc-3ab4952f4dc6" (UID: "2835390b-006c-4fc4-81fc-3ab4952f4dc6"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:01.544232 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.544191 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2835390b-006c-4fc4-81fc-3ab4952f4dc6-kube-api-access-qpgwh" pod "2835390b-006c-4fc4-81fc-3ab4952f4dc6" (UID: "2835390b-006c-4fc4-81fc-3ab4952f4dc6"). InnerVolumeSpecName "kube-api-access-qpgwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:34:01.557974 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.557459 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:34:01.557974 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.557737 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="4c82f811-e9e7-4c22-ba2b-83e9419752ed" containerName="alertmanager" containerID="cri-o://81761ef6c91ecf83242e65ed237a1c6747f899843ace3fd6af253290b8d204c3" gracePeriod=120 Oct 30 11:34:01.557974 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.557893 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="4c82f811-e9e7-4c22-ba2b-83e9419752ed" containerName="config-reloader" containerID="cri-o://fab688ee088d845b5a234e36b02d38d08d033d600d280016f6e5ac74b995905b" gracePeriod=120 Oct 30 11:34:01.562416 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.562387 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 11:34:01.562951 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.562820 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="fffff5fd-6828-41bf-88aa-d0e721fedbf0" containerName="config-reloader" containerID="cri-o://fafd07d8121393996f64ebec92eeb3b54c2ffe2f52913f42bc9e058b3a5047e6" gracePeriod=600 Oct 30 11:34:01.562951 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.562830 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="fffff5fd-6828-41bf-88aa-d0e721fedbf0" containerName="thanos-sidecar" containerID="cri-o://65195ead31a8723aa11b20f1e70b1604acb5d2af76798261fc58d4a5f39cfc5e" gracePeriod=600 Oct 30 11:34:01.564839 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.562756 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="fffff5fd-6828-41bf-88aa-d0e721fedbf0" containerName="prometheus" containerID="cri-o://b0e5c885df440ef973a805a54acb26e3335937f8893c81ce3589e2afe49b7232" gracePeriod=600 Oct 30 11:34:01.639278 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.639186 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2835390b-006c-4fc4-81fc-3ab4952f4dc6-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:01.639278 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.639208 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpgwh\" (UniqueName: \"kubernetes.io/projected/2835390b-006c-4fc4-81fc-3ab4952f4dc6-kube-api-access-qpgwh\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:01.639278 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.639223 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2835390b-006c-4fc4-81fc-3ab4952f4dc6-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:01.683054 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.683012 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:01.708981 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.708938 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="fffff5fd-6828-41bf-88aa-d0e721fedbf0" containerName="prometheus" probeResult="failure" output="Get \"https://10.134.0.116:9090/-/ready\": dial tcp 10.134.0.116:9090: connect: connection refused" Oct 30 11:34:01.763721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.763685 4289 generic.go:356] "Generic (PLEG): container finished" podID="2835390b-006c-4fc4-81fc-3ab4952f4dc6" containerID="ab6ddb4cf96c5fc5e978fe0fb7f028a7f0e8234d64601a77c0885e86e0a8a7cb" exitCode=0 Oct 30 11:34:01.763721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.763718 4289 generic.go:356] "Generic (PLEG): container finished" podID="2835390b-006c-4fc4-81fc-3ab4952f4dc6" containerID="b98e3e5a6947742dce73f2a9df655696e64ea64e3905dc583d1213ce58eee593" exitCode=0 Oct 30 11:34:01.763721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.763731 4289 generic.go:356] "Generic (PLEG): container finished" podID="2835390b-006c-4fc4-81fc-3ab4952f4dc6" containerID="f32449831bce2e1f1761659b07c517f700bd73ed9549d59cb1e84be9db5edb7c" exitCode=0 Oct 30 11:34:01.764066 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.763739 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" Oct 30 11:34:01.764066 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.763762 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" event={"ID":"2835390b-006c-4fc4-81fc-3ab4952f4dc6","Type":"ContainerDied","Data":"ab6ddb4cf96c5fc5e978fe0fb7f028a7f0e8234d64601a77c0885e86e0a8a7cb"} Oct 30 11:34:01.764066 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.763834 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" event={"ID":"2835390b-006c-4fc4-81fc-3ab4952f4dc6","Type":"ContainerDied","Data":"b98e3e5a6947742dce73f2a9df655696e64ea64e3905dc583d1213ce58eee593"} Oct 30 11:34:01.764066 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.763886 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" event={"ID":"2835390b-006c-4fc4-81fc-3ab4952f4dc6","Type":"ContainerDied","Data":"f32449831bce2e1f1761659b07c517f700bd73ed9549d59cb1e84be9db5edb7c"} Oct 30 11:34:01.764066 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.763916 4289 scope.go:122] "RemoveContainer" containerID="ab6ddb4cf96c5fc5e978fe0fb7f028a7f0e8234d64601a77c0885e86e0a8a7cb" Oct 30 11:34:01.764066 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.763924 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48" event={"ID":"2835390b-006c-4fc4-81fc-3ab4952f4dc6","Type":"ContainerDied","Data":"78f214586f070210c68ed6c58fd7b0940bb7ca9012135d9e0ca57939dda52ecb"} Oct 30 11:34:01.765458 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.765411 4289 generic.go:356] "Generic (PLEG): container finished" podID="098dce60-e115-4404-8e5f-43809c74e6b6" containerID="9684d7378a6e5ca8719fcfcfa5cd9b75a47821655099c926e72b62c8e043e3c6" exitCode=0 Oct 30 11:34:01.765568 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.765487 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg" Oct 30 11:34:01.765631 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.765472 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg" event={"ID":"098dce60-e115-4404-8e5f-43809c74e6b6","Type":"ContainerDied","Data":"9684d7378a6e5ca8719fcfcfa5cd9b75a47821655099c926e72b62c8e043e3c6"} Oct 30 11:34:01.765631 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.765619 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg" event={"ID":"098dce60-e115-4404-8e5f-43809c74e6b6","Type":"ContainerDied","Data":"b3db4a60bd9c6dd2c16c429db500ea52453160a6948bdcae2a64e559daae1827"} Oct 30 11:34:01.767410 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.767386 4289 generic.go:356] "Generic (PLEG): container finished" podID="86e3cacb-a113-4182-b955-fd45c0e77190" containerID="b8c4a519d62c6f7b6f5be62b1c0f7d31ee31ea9bed6db61f9c8bf2531ee224c9" exitCode=0 Oct 30 11:34:01.767549 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.767452 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" Oct 30 11:34:01.767549 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.767480 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" event={"ID":"86e3cacb-a113-4182-b955-fd45c0e77190","Type":"ContainerDied","Data":"b8c4a519d62c6f7b6f5be62b1c0f7d31ee31ea9bed6db61f9c8bf2531ee224c9"} Oct 30 11:34:01.767549 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.767538 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7" event={"ID":"86e3cacb-a113-4182-b955-fd45c0e77190","Type":"ContainerDied","Data":"3b054e557bd115e0401eb6963376e09b563b0e4c8dc7d7aa08bb3f531845d5be"} Oct 30 11:34:01.769725 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.769703 4289 generic.go:356] "Generic (PLEG): container finished" podID="17a81880-2dca-4878-bf5a-6f7185f4451e" containerID="7e5b38c85078f1929b906b4de6d76e82b7fd55b8323d43ce5fc2c9bb7d4e509a" exitCode=0 Oct 30 11:34:01.769725 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.769720 4289 generic.go:356] "Generic (PLEG): container finished" podID="17a81880-2dca-4878-bf5a-6f7185f4451e" containerID="ee721e3921d2ae3ec4c2d6be3fe151ca597cb50375ba9aa82d914664aec59fec" exitCode=0 Oct 30 11:34:01.769901 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.769730 4289 generic.go:356] "Generic (PLEG): container finished" podID="17a81880-2dca-4878-bf5a-6f7185f4451e" containerID="cf4b519ce878052d3160b1237873c33573f01335ffb9abf6c71eea160bfcfdb4" exitCode=0 Oct 30 11:34:01.769901 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.769763 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" Oct 30 11:34:01.770024 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.769762 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" event={"ID":"17a81880-2dca-4878-bf5a-6f7185f4451e","Type":"ContainerDied","Data":"7e5b38c85078f1929b906b4de6d76e82b7fd55b8323d43ce5fc2c9bb7d4e509a"} Oct 30 11:34:01.770024 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.769971 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" event={"ID":"17a81880-2dca-4878-bf5a-6f7185f4451e","Type":"ContainerDied","Data":"ee721e3921d2ae3ec4c2d6be3fe151ca597cb50375ba9aa82d914664aec59fec"} Oct 30 11:34:01.770024 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.770019 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" event={"ID":"17a81880-2dca-4878-bf5a-6f7185f4451e","Type":"ContainerDied","Data":"cf4b519ce878052d3160b1237873c33573f01335ffb9abf6c71eea160bfcfdb4"} Oct 30 11:34:01.770191 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.770059 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2" event={"ID":"17a81880-2dca-4878-bf5a-6f7185f4451e","Type":"ContainerDied","Data":"aee5826d7f0f4cbae9064a68e26c82b9c9da6dcebe3486916dd5045cd2466c09"} Oct 30 11:34:01.774553 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.774531 4289 generic.go:356] "Generic (PLEG): container finished" podID="4c82f811-e9e7-4c22-ba2b-83e9419752ed" containerID="fab688ee088d845b5a234e36b02d38d08d033d600d280016f6e5ac74b995905b" exitCode=0 Oct 30 11:34:01.774680 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.774666 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"4c82f811-e9e7-4c22-ba2b-83e9419752ed","Type":"ContainerDied","Data":"fab688ee088d845b5a234e36b02d38d08d033d600d280016f6e5ac74b995905b"} Oct 30 11:34:01.776134 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.776062 4289 scope.go:122] "RemoveContainer" containerID="b98e3e5a6947742dce73f2a9df655696e64ea64e3905dc583d1213ce58eee593" Oct 30 11:34:01.777501 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.777480 4289 generic.go:356] "Generic (PLEG): container finished" podID="fffff5fd-6828-41bf-88aa-d0e721fedbf0" containerID="65195ead31a8723aa11b20f1e70b1604acb5d2af76798261fc58d4a5f39cfc5e" exitCode=0 Oct 30 11:34:01.777501 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.777499 4289 generic.go:356] "Generic (PLEG): container finished" podID="fffff5fd-6828-41bf-88aa-d0e721fedbf0" containerID="fafd07d8121393996f64ebec92eeb3b54c2ffe2f52913f42bc9e058b3a5047e6" exitCode=0 Oct 30 11:34:01.777740 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.777505 4289 generic.go:356] "Generic (PLEG): container finished" podID="fffff5fd-6828-41bf-88aa-d0e721fedbf0" containerID="b0e5c885df440ef973a805a54acb26e3335937f8893c81ce3589e2afe49b7232" exitCode=0 Oct 30 11:34:01.777740 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.777526 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fffff5fd-6828-41bf-88aa-d0e721fedbf0","Type":"ContainerDied","Data":"65195ead31a8723aa11b20f1e70b1604acb5d2af76798261fc58d4a5f39cfc5e"} Oct 30 11:34:01.777740 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.777574 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fffff5fd-6828-41bf-88aa-d0e721fedbf0","Type":"ContainerDied","Data":"fafd07d8121393996f64ebec92eeb3b54c2ffe2f52913f42bc9e058b3a5047e6"} Oct 30 11:34:01.777740 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.777606 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fffff5fd-6828-41bf-88aa-d0e721fedbf0","Type":"ContainerDied","Data":"b0e5c885df440ef973a805a54acb26e3335937f8893c81ce3589e2afe49b7232"} Oct 30 11:34:01.784205 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.784178 4289 scope.go:122] "RemoveContainer" containerID="f32449831bce2e1f1761659b07c517f700bd73ed9549d59cb1e84be9db5edb7c" Oct 30 11:34:01.791973 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.791947 4289 scope.go:122] "RemoveContainer" containerID="ab6ddb4cf96c5fc5e978fe0fb7f028a7f0e8234d64601a77c0885e86e0a8a7cb" Oct 30 11:34:01.792249 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:01.792233 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab6ddb4cf96c5fc5e978fe0fb7f028a7f0e8234d64601a77c0885e86e0a8a7cb\": container with ID starting with ab6ddb4cf96c5fc5e978fe0fb7f028a7f0e8234d64601a77c0885e86e0a8a7cb not found: ID does not exist" containerID="ab6ddb4cf96c5fc5e978fe0fb7f028a7f0e8234d64601a77c0885e86e0a8a7cb" Oct 30 11:34:01.792300 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.792262 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab6ddb4cf96c5fc5e978fe0fb7f028a7f0e8234d64601a77c0885e86e0a8a7cb"} err="failed to get container status \"ab6ddb4cf96c5fc5e978fe0fb7f028a7f0e8234d64601a77c0885e86e0a8a7cb\": rpc error: code = NotFound desc = could not find container \"ab6ddb4cf96c5fc5e978fe0fb7f028a7f0e8234d64601a77c0885e86e0a8a7cb\": container with ID starting with ab6ddb4cf96c5fc5e978fe0fb7f028a7f0e8234d64601a77c0885e86e0a8a7cb not found: ID does not exist" Oct 30 11:34:01.792300 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.792280 4289 scope.go:122] "RemoveContainer" containerID="b98e3e5a6947742dce73f2a9df655696e64ea64e3905dc583d1213ce58eee593" Oct 30 11:34:01.792553 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:01.792535 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b98e3e5a6947742dce73f2a9df655696e64ea64e3905dc583d1213ce58eee593\": container with ID starting with b98e3e5a6947742dce73f2a9df655696e64ea64e3905dc583d1213ce58eee593 not found: ID does not exist" containerID="b98e3e5a6947742dce73f2a9df655696e64ea64e3905dc583d1213ce58eee593" Oct 30 11:34:01.792618 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.792559 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98e3e5a6947742dce73f2a9df655696e64ea64e3905dc583d1213ce58eee593"} err="failed to get container status \"b98e3e5a6947742dce73f2a9df655696e64ea64e3905dc583d1213ce58eee593\": rpc error: code = NotFound desc = could not find container \"b98e3e5a6947742dce73f2a9df655696e64ea64e3905dc583d1213ce58eee593\": container with ID starting with b98e3e5a6947742dce73f2a9df655696e64ea64e3905dc583d1213ce58eee593 not found: ID does not exist" Oct 30 11:34:01.792618 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.792581 4289 scope.go:122] "RemoveContainer" containerID="f32449831bce2e1f1761659b07c517f700bd73ed9549d59cb1e84be9db5edb7c" Oct 30 11:34:01.792835 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:01.792820 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f32449831bce2e1f1761659b07c517f700bd73ed9549d59cb1e84be9db5edb7c\": container with ID starting with f32449831bce2e1f1761659b07c517f700bd73ed9549d59cb1e84be9db5edb7c not found: ID does not exist" containerID="f32449831bce2e1f1761659b07c517f700bd73ed9549d59cb1e84be9db5edb7c" Oct 30 11:34:01.792900 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.792839 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f32449831bce2e1f1761659b07c517f700bd73ed9549d59cb1e84be9db5edb7c"} err="failed to get container status \"f32449831bce2e1f1761659b07c517f700bd73ed9549d59cb1e84be9db5edb7c\": rpc error: code = NotFound desc = could not find container \"f32449831bce2e1f1761659b07c517f700bd73ed9549d59cb1e84be9db5edb7c\": container with ID starting with f32449831bce2e1f1761659b07c517f700bd73ed9549d59cb1e84be9db5edb7c not found: ID does not exist" Oct 30 11:34:01.792900 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.792855 4289 scope.go:122] "RemoveContainer" containerID="ab6ddb4cf96c5fc5e978fe0fb7f028a7f0e8234d64601a77c0885e86e0a8a7cb" Oct 30 11:34:01.793091 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.793068 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab6ddb4cf96c5fc5e978fe0fb7f028a7f0e8234d64601a77c0885e86e0a8a7cb"} err="failed to get container status \"ab6ddb4cf96c5fc5e978fe0fb7f028a7f0e8234d64601a77c0885e86e0a8a7cb\": rpc error: code = NotFound desc = could not find container \"ab6ddb4cf96c5fc5e978fe0fb7f028a7f0e8234d64601a77c0885e86e0a8a7cb\": container with ID starting with ab6ddb4cf96c5fc5e978fe0fb7f028a7f0e8234d64601a77c0885e86e0a8a7cb not found: ID does not exist" Oct 30 11:34:01.793146 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.793093 4289 scope.go:122] "RemoveContainer" containerID="b98e3e5a6947742dce73f2a9df655696e64ea64e3905dc583d1213ce58eee593" Oct 30 11:34:01.793356 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.793336 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98e3e5a6947742dce73f2a9df655696e64ea64e3905dc583d1213ce58eee593"} err="failed to get container status \"b98e3e5a6947742dce73f2a9df655696e64ea64e3905dc583d1213ce58eee593\": rpc error: code = NotFound desc = could not find container \"b98e3e5a6947742dce73f2a9df655696e64ea64e3905dc583d1213ce58eee593\": container with ID starting with b98e3e5a6947742dce73f2a9df655696e64ea64e3905dc583d1213ce58eee593 not found: ID does not exist" Oct 30 11:34:01.793356 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.793357 4289 scope.go:122] "RemoveContainer" containerID="f32449831bce2e1f1761659b07c517f700bd73ed9549d59cb1e84be9db5edb7c" Oct 30 11:34:01.793663 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.793642 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f32449831bce2e1f1761659b07c517f700bd73ed9549d59cb1e84be9db5edb7c"} err="failed to get container status \"f32449831bce2e1f1761659b07c517f700bd73ed9549d59cb1e84be9db5edb7c\": rpc error: code = NotFound desc = could not find container \"f32449831bce2e1f1761659b07c517f700bd73ed9549d59cb1e84be9db5edb7c\": container with ID starting with f32449831bce2e1f1761659b07c517f700bd73ed9549d59cb1e84be9db5edb7c not found: ID does not exist" Oct 30 11:34:01.793752 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.793664 4289 scope.go:122] "RemoveContainer" containerID="ab6ddb4cf96c5fc5e978fe0fb7f028a7f0e8234d64601a77c0885e86e0a8a7cb" Oct 30 11:34:01.793932 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.793914 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab6ddb4cf96c5fc5e978fe0fb7f028a7f0e8234d64601a77c0885e86e0a8a7cb"} err="failed to get container status \"ab6ddb4cf96c5fc5e978fe0fb7f028a7f0e8234d64601a77c0885e86e0a8a7cb\": rpc error: code = NotFound desc = could not find container \"ab6ddb4cf96c5fc5e978fe0fb7f028a7f0e8234d64601a77c0885e86e0a8a7cb\": container with ID starting with ab6ddb4cf96c5fc5e978fe0fb7f028a7f0e8234d64601a77c0885e86e0a8a7cb not found: ID does not exist" Oct 30 11:34:01.794007 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.793932 4289 scope.go:122] "RemoveContainer" containerID="b98e3e5a6947742dce73f2a9df655696e64ea64e3905dc583d1213ce58eee593" Oct 30 11:34:01.794156 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.794135 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98e3e5a6947742dce73f2a9df655696e64ea64e3905dc583d1213ce58eee593"} err="failed to get container status \"b98e3e5a6947742dce73f2a9df655696e64ea64e3905dc583d1213ce58eee593\": rpc error: code = NotFound desc = could not find container \"b98e3e5a6947742dce73f2a9df655696e64ea64e3905dc583d1213ce58eee593\": container with ID starting with b98e3e5a6947742dce73f2a9df655696e64ea64e3905dc583d1213ce58eee593 not found: ID does not exist" Oct 30 11:34:01.794156 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.794156 4289 scope.go:122] "RemoveContainer" containerID="f32449831bce2e1f1761659b07c517f700bd73ed9549d59cb1e84be9db5edb7c" Oct 30 11:34:01.794484 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.794465 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f32449831bce2e1f1761659b07c517f700bd73ed9549d59cb1e84be9db5edb7c"} err="failed to get container status \"f32449831bce2e1f1761659b07c517f700bd73ed9549d59cb1e84be9db5edb7c\": rpc error: code = NotFound desc = could not find container \"f32449831bce2e1f1761659b07c517f700bd73ed9549d59cb1e84be9db5edb7c\": container with ID starting with f32449831bce2e1f1761659b07c517f700bd73ed9549d59cb1e84be9db5edb7c not found: ID does not exist" Oct 30 11:34:01.794484 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.794483 4289 scope.go:122] "RemoveContainer" containerID="9684d7378a6e5ca8719fcfcfa5cd9b75a47821655099c926e72b62c8e043e3c6" Oct 30 11:34:01.802157 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.802143 4289 scope.go:122] "RemoveContainer" containerID="9684d7378a6e5ca8719fcfcfa5cd9b75a47821655099c926e72b62c8e043e3c6" Oct 30 11:34:01.802418 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:01.802395 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9684d7378a6e5ca8719fcfcfa5cd9b75a47821655099c926e72b62c8e043e3c6\": container with ID starting with 9684d7378a6e5ca8719fcfcfa5cd9b75a47821655099c926e72b62c8e043e3c6 not found: ID does not exist" containerID="9684d7378a6e5ca8719fcfcfa5cd9b75a47821655099c926e72b62c8e043e3c6" Oct 30 11:34:01.802542 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.802420 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9684d7378a6e5ca8719fcfcfa5cd9b75a47821655099c926e72b62c8e043e3c6"} err="failed to get container status \"9684d7378a6e5ca8719fcfcfa5cd9b75a47821655099c926e72b62c8e043e3c6\": rpc error: code = NotFound desc = could not find container \"9684d7378a6e5ca8719fcfcfa5cd9b75a47821655099c926e72b62c8e043e3c6\": container with ID starting with 9684d7378a6e5ca8719fcfcfa5cd9b75a47821655099c926e72b62c8e043e3c6 not found: ID does not exist" Oct 30 11:34:01.802542 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.802447 4289 scope.go:122] "RemoveContainer" containerID="b8c4a519d62c6f7b6f5be62b1c0f7d31ee31ea9bed6db61f9c8bf2531ee224c9" Oct 30 11:34:01.810285 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.810266 4289 scope.go:122] "RemoveContainer" containerID="b8c4a519d62c6f7b6f5be62b1c0f7d31ee31ea9bed6db61f9c8bf2531ee224c9" Oct 30 11:34:01.810582 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:01.810563 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8c4a519d62c6f7b6f5be62b1c0f7d31ee31ea9bed6db61f9c8bf2531ee224c9\": container with ID starting with b8c4a519d62c6f7b6f5be62b1c0f7d31ee31ea9bed6db61f9c8bf2531ee224c9 not found: ID does not exist" containerID="b8c4a519d62c6f7b6f5be62b1c0f7d31ee31ea9bed6db61f9c8bf2531ee224c9" Oct 30 11:34:01.810659 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.810588 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c4a519d62c6f7b6f5be62b1c0f7d31ee31ea9bed6db61f9c8bf2531ee224c9"} err="failed to get container status \"b8c4a519d62c6f7b6f5be62b1c0f7d31ee31ea9bed6db61f9c8bf2531ee224c9\": rpc error: code = NotFound desc = could not find container \"b8c4a519d62c6f7b6f5be62b1c0f7d31ee31ea9bed6db61f9c8bf2531ee224c9\": container with ID starting with b8c4a519d62c6f7b6f5be62b1c0f7d31ee31ea9bed6db61f9c8bf2531ee224c9 not found: ID does not exist" Oct 30 11:34:01.810659 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.810603 4289 scope.go:122] "RemoveContainer" containerID="7e5b38c85078f1929b906b4de6d76e82b7fd55b8323d43ce5fc2c9bb7d4e509a" Oct 30 11:34:01.816364 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.816341 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7"] Oct 30 11:34:01.818422 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.818403 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:34:01.819517 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.819502 4289 scope.go:122] "RemoveContainer" containerID="ee721e3921d2ae3ec4c2d6be3fe151ca597cb50375ba9aa82d914664aec59fec" Oct 30 11:34:01.820255 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.820210 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-8v5d7"] Oct 30 11:34:01.826460 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.826440 4289 scope.go:122] "RemoveContainer" containerID="cf4b519ce878052d3160b1237873c33573f01335ffb9abf6c71eea160bfcfdb4" Oct 30 11:34:01.833408 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.833390 4289 scope.go:122] "RemoveContainer" containerID="7e5b38c85078f1929b906b4de6d76e82b7fd55b8323d43ce5fc2c9bb7d4e509a" Oct 30 11:34:01.833760 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:01.833744 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e5b38c85078f1929b906b4de6d76e82b7fd55b8323d43ce5fc2c9bb7d4e509a\": container with ID starting with 7e5b38c85078f1929b906b4de6d76e82b7fd55b8323d43ce5fc2c9bb7d4e509a not found: ID does not exist" containerID="7e5b38c85078f1929b906b4de6d76e82b7fd55b8323d43ce5fc2c9bb7d4e509a" Oct 30 11:34:01.833824 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.833774 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e5b38c85078f1929b906b4de6d76e82b7fd55b8323d43ce5fc2c9bb7d4e509a"} err="failed to get container status \"7e5b38c85078f1929b906b4de6d76e82b7fd55b8323d43ce5fc2c9bb7d4e509a\": rpc error: code = NotFound desc = could not find container \"7e5b38c85078f1929b906b4de6d76e82b7fd55b8323d43ce5fc2c9bb7d4e509a\": container with ID starting with 7e5b38c85078f1929b906b4de6d76e82b7fd55b8323d43ce5fc2c9bb7d4e509a not found: ID does not exist" Oct 30 11:34:01.833824 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.833789 4289 scope.go:122] "RemoveContainer" containerID="ee721e3921d2ae3ec4c2d6be3fe151ca597cb50375ba9aa82d914664aec59fec" Oct 30 11:34:01.833994 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:01.833978 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee721e3921d2ae3ec4c2d6be3fe151ca597cb50375ba9aa82d914664aec59fec\": container with ID starting with ee721e3921d2ae3ec4c2d6be3fe151ca597cb50375ba9aa82d914664aec59fec not found: ID does not exist" containerID="ee721e3921d2ae3ec4c2d6be3fe151ca597cb50375ba9aa82d914664aec59fec" Oct 30 11:34:01.834029 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.833998 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee721e3921d2ae3ec4c2d6be3fe151ca597cb50375ba9aa82d914664aec59fec"} err="failed to get container status \"ee721e3921d2ae3ec4c2d6be3fe151ca597cb50375ba9aa82d914664aec59fec\": rpc error: code = NotFound desc = could not find container \"ee721e3921d2ae3ec4c2d6be3fe151ca597cb50375ba9aa82d914664aec59fec\": container with ID starting with ee721e3921d2ae3ec4c2d6be3fe151ca597cb50375ba9aa82d914664aec59fec not found: ID does not exist" Oct 30 11:34:01.834029 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.834011 4289 scope.go:122] "RemoveContainer" containerID="cf4b519ce878052d3160b1237873c33573f01335ffb9abf6c71eea160bfcfdb4" Oct 30 11:34:01.834208 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:01.834191 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf4b519ce878052d3160b1237873c33573f01335ffb9abf6c71eea160bfcfdb4\": container with ID starting with cf4b519ce878052d3160b1237873c33573f01335ffb9abf6c71eea160bfcfdb4 not found: ID does not exist" containerID="cf4b519ce878052d3160b1237873c33573f01335ffb9abf6c71eea160bfcfdb4" Oct 30 11:34:01.834255 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.834210 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf4b519ce878052d3160b1237873c33573f01335ffb9abf6c71eea160bfcfdb4"} err="failed to get container status \"cf4b519ce878052d3160b1237873c33573f01335ffb9abf6c71eea160bfcfdb4\": rpc error: code = NotFound desc = could not find container \"cf4b519ce878052d3160b1237873c33573f01335ffb9abf6c71eea160bfcfdb4\": container with ID starting with cf4b519ce878052d3160b1237873c33573f01335ffb9abf6c71eea160bfcfdb4 not found: ID does not exist" Oct 30 11:34:01.834255 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.834224 4289 scope.go:122] "RemoveContainer" containerID="7e5b38c85078f1929b906b4de6d76e82b7fd55b8323d43ce5fc2c9bb7d4e509a" Oct 30 11:34:01.834467 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.834423 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e5b38c85078f1929b906b4de6d76e82b7fd55b8323d43ce5fc2c9bb7d4e509a"} err="failed to get container status \"7e5b38c85078f1929b906b4de6d76e82b7fd55b8323d43ce5fc2c9bb7d4e509a\": rpc error: code = NotFound desc = could not find container \"7e5b38c85078f1929b906b4de6d76e82b7fd55b8323d43ce5fc2c9bb7d4e509a\": container with ID starting with 7e5b38c85078f1929b906b4de6d76e82b7fd55b8323d43ce5fc2c9bb7d4e509a not found: ID does not exist" Oct 30 11:34:01.834467 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.834466 4289 scope.go:122] "RemoveContainer" containerID="ee721e3921d2ae3ec4c2d6be3fe151ca597cb50375ba9aa82d914664aec59fec" Oct 30 11:34:01.834691 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.834670 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee721e3921d2ae3ec4c2d6be3fe151ca597cb50375ba9aa82d914664aec59fec"} err="failed to get container status \"ee721e3921d2ae3ec4c2d6be3fe151ca597cb50375ba9aa82d914664aec59fec\": rpc error: code = NotFound desc = could not find container \"ee721e3921d2ae3ec4c2d6be3fe151ca597cb50375ba9aa82d914664aec59fec\": container with ID starting with ee721e3921d2ae3ec4c2d6be3fe151ca597cb50375ba9aa82d914664aec59fec not found: ID does not exist" Oct 30 11:34:01.834764 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.834692 4289 scope.go:122] "RemoveContainer" containerID="cf4b519ce878052d3160b1237873c33573f01335ffb9abf6c71eea160bfcfdb4" Oct 30 11:34:01.834921 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.834901 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf4b519ce878052d3160b1237873c33573f01335ffb9abf6c71eea160bfcfdb4"} err="failed to get container status \"cf4b519ce878052d3160b1237873c33573f01335ffb9abf6c71eea160bfcfdb4\": rpc error: code = NotFound desc = could not find container \"cf4b519ce878052d3160b1237873c33573f01335ffb9abf6c71eea160bfcfdb4\": container with ID starting with cf4b519ce878052d3160b1237873c33573f01335ffb9abf6c71eea160bfcfdb4 not found: ID does not exist" Oct 30 11:34:01.834971 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.834921 4289 scope.go:122] "RemoveContainer" containerID="7e5b38c85078f1929b906b4de6d76e82b7fd55b8323d43ce5fc2c9bb7d4e509a" Oct 30 11:34:01.835154 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.835138 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e5b38c85078f1929b906b4de6d76e82b7fd55b8323d43ce5fc2c9bb7d4e509a"} err="failed to get container status \"7e5b38c85078f1929b906b4de6d76e82b7fd55b8323d43ce5fc2c9bb7d4e509a\": rpc error: code = NotFound desc = could not find container \"7e5b38c85078f1929b906b4de6d76e82b7fd55b8323d43ce5fc2c9bb7d4e509a\": container with ID starting with 7e5b38c85078f1929b906b4de6d76e82b7fd55b8323d43ce5fc2c9bb7d4e509a not found: ID does not exist" Oct 30 11:34:01.835225 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.835158 4289 scope.go:122] "RemoveContainer" containerID="ee721e3921d2ae3ec4c2d6be3fe151ca597cb50375ba9aa82d914664aec59fec" Oct 30 11:34:01.835376 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.835358 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee721e3921d2ae3ec4c2d6be3fe151ca597cb50375ba9aa82d914664aec59fec"} err="failed to get container status \"ee721e3921d2ae3ec4c2d6be3fe151ca597cb50375ba9aa82d914664aec59fec\": rpc error: code = NotFound desc = could not find container \"ee721e3921d2ae3ec4c2d6be3fe151ca597cb50375ba9aa82d914664aec59fec\": container with ID starting with ee721e3921d2ae3ec4c2d6be3fe151ca597cb50375ba9aa82d914664aec59fec not found: ID does not exist" Oct 30 11:34:01.835422 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.835378 4289 scope.go:122] "RemoveContainer" containerID="cf4b519ce878052d3160b1237873c33573f01335ffb9abf6c71eea160bfcfdb4" Oct 30 11:34:01.835613 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.835595 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf4b519ce878052d3160b1237873c33573f01335ffb9abf6c71eea160bfcfdb4"} err="failed to get container status \"cf4b519ce878052d3160b1237873c33573f01335ffb9abf6c71eea160bfcfdb4\": rpc error: code = NotFound desc = could not find container \"cf4b519ce878052d3160b1237873c33573f01335ffb9abf6c71eea160bfcfdb4\": container with ID starting with cf4b519ce878052d3160b1237873c33573f01335ffb9abf6c71eea160bfcfdb4 not found: ID does not exist" Oct 30 11:34:01.853164 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.853141 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg"] Oct 30 11:34:01.854201 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.854169 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:01.858034 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.857993 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-xvlpg"] Oct 30 11:34:01.890560 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.890527 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2"] Oct 30 11:34:01.894182 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.894152 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-d2jm2"] Oct 30 11:34:01.930837 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.930808 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48"] Oct 30 11:34:01.934735 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.934710 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-hmd48"] Oct 30 11:34:01.941710 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.941688 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fffff5fd-6828-41bf-88aa-d0e721fedbf0-kube-api-access-288n6\" (UniqueName: \"kubernetes.io/projected/fffff5fd-6828-41bf-88aa-d0e721fedbf0-kube-api-access-288n6\") pod \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " Oct 30 11:34:01.941838 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.941825 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396\") pod \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " Oct 30 11:34:01.941887 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.941854 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " Oct 30 11:34:01.941887 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.941879 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fffff5fd-6828-41bf-88aa-d0e721fedbf0-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fffff5fd-6828-41bf-88aa-d0e721fedbf0-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " Oct 30 11:34:01.941955 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.941898 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fffff5fd-6828-41bf-88aa-d0e721fedbf0-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fffff5fd-6828-41bf-88aa-d0e721fedbf0-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " Oct 30 11:34:01.941955 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.941940 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-web-config\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-web-config\") pod \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " Oct 30 11:34:01.942065 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.941981 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-thanos-prometheus-http-client-file\") pod \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " Oct 30 11:34:01.942065 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.942019 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-secret-prometheus-web-tls-ca\") pod \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " Oct 30 11:34:01.942179 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.942083 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffff5fd-6828-41bf-88aa-d0e721fedbf0-config-out\" (UniqueName: \"kubernetes.io/empty-dir/fffff5fd-6828-41bf-88aa-d0e721fedbf0-config-out\") pod \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " Oct 30 11:34:01.942179 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.942133 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-config\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-config\") pod \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " Oct 30 11:34:01.942288 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.942178 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " Oct 30 11:34:01.942288 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.942208 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fffff5fd-6828-41bf-88aa-d0e721fedbf0-tls-assets\" (UniqueName: \"kubernetes.io/projected/fffff5fd-6828-41bf-88aa-d0e721fedbf0-tls-assets\") pod \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " Oct 30 11:34:01.942288 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.942242 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fffff5fd-6828-41bf-88aa-d0e721fedbf0-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fffff5fd-6828-41bf-88aa-d0e721fedbf0-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\" (UID: \"fffff5fd-6828-41bf-88aa-d0e721fedbf0\") " Oct 30 11:34:01.942288 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.942260 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffff5fd-6828-41bf-88aa-d0e721fedbf0-prometheus-data-science-monitoringstack-rulefiles-1" pod "fffff5fd-6828-41bf-88aa-d0e721fedbf0" (UID: "fffff5fd-6828-41bf-88aa-d0e721fedbf0"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:34:01.942533 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.942336 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffff5fd-6828-41bf-88aa-d0e721fedbf0-prometheus-data-science-monitoringstack-rulefiles-0" pod "fffff5fd-6828-41bf-88aa-d0e721fedbf0" (UID: "fffff5fd-6828-41bf-88aa-d0e721fedbf0"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:34:01.942600 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.942578 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fffff5fd-6828-41bf-88aa-d0e721fedbf0-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:01.942665 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.942600 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fffff5fd-6828-41bf-88aa-d0e721fedbf0-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:01.942975 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.942930 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffff5fd-6828-41bf-88aa-d0e721fedbf0-prometheus-data-science-monitoringstack-rulefiles-2" pod "fffff5fd-6828-41bf-88aa-d0e721fedbf0" (UID: "fffff5fd-6828-41bf-88aa-d0e721fedbf0"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:34:01.944824 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.944535 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "fffff5fd-6828-41bf-88aa-d0e721fedbf0" (UID: "fffff5fd-6828-41bf-88aa-d0e721fedbf0"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:01.944824 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.944624 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-config" pod "fffff5fd-6828-41bf-88aa-d0e721fedbf0" (UID: "fffff5fd-6828-41bf-88aa-d0e721fedbf0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:01.944824 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.944784 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-thanos-prometheus-http-client-file" pod "fffff5fd-6828-41bf-88aa-d0e721fedbf0" (UID: "fffff5fd-6828-41bf-88aa-d0e721fedbf0"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:01.945069 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.944829 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffff5fd-6828-41bf-88aa-d0e721fedbf0-kube-api-access-288n6" pod "fffff5fd-6828-41bf-88aa-d0e721fedbf0" (UID: "fffff5fd-6828-41bf-88aa-d0e721fedbf0"). InnerVolumeSpecName "kube-api-access-288n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:34:01.945069 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.945021 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffff5fd-6828-41bf-88aa-d0e721fedbf0-tls-assets" pod "fffff5fd-6828-41bf-88aa-d0e721fedbf0" (UID: "fffff5fd-6828-41bf-88aa-d0e721fedbf0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:34:01.945850 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.945290 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-secret-prometheus-web-tls-ca" pod "fffff5fd-6828-41bf-88aa-d0e721fedbf0" (UID: "fffff5fd-6828-41bf-88aa-d0e721fedbf0"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:01.946543 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.946514 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffff5fd-6828-41bf-88aa-d0e721fedbf0-config-out" pod "fffff5fd-6828-41bf-88aa-d0e721fedbf0" (UID: "fffff5fd-6828-41bf-88aa-d0e721fedbf0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:34:01.946758 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.946733 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "fffff5fd-6828-41bf-88aa-d0e721fedbf0" (UID: "fffff5fd-6828-41bf-88aa-d0e721fedbf0"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:01.947337 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.947316 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396" pod "fffff5fd-6828-41bf-88aa-d0e721fedbf0" (UID: "fffff5fd-6828-41bf-88aa-d0e721fedbf0"). InnerVolumeSpecName "pvc-ae2fc6cc-1a29-41df-b109-2b8489d4e098". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 11:34:01.957148 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:01.957125 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-web-config" pod "fffff5fd-6828-41bf-88aa-d0e721fedbf0" (UID: "fffff5fd-6828-41bf-88aa-d0e721fedbf0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:02.020320 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.020281 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:02.043448 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.043405 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fffff5fd-6828-41bf-88aa-d0e721fedbf0-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:02.043869 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.043470 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-288n6\" (UniqueName: \"kubernetes.io/projected/fffff5fd-6828-41bf-88aa-d0e721fedbf0-kube-api-access-288n6\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:02.043869 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.043511 4289 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-ae2fc6cc-1a29-41df-b109-2b8489d4e098\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396\") on node \"ip-10-0-137-242.ec2.internal\" " Oct 30 11:34:02.043869 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.043528 4289 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:02.043869 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.043544 4289 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-web-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:02.043869 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.043560 4289 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:02.043869 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.043575 4289 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-secret-prometheus-web-tls-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:02.043869 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.043591 4289 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fffff5fd-6828-41bf-88aa-d0e721fedbf0-config-out\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:02.043869 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.043605 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:02.043869 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.043619 4289 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fffff5fd-6828-41bf-88aa-d0e721fedbf0-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:02.043869 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.043635 4289 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fffff5fd-6828-41bf-88aa-d0e721fedbf0-tls-assets\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:02.056234 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.056216 4289 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-ae2fc6cc-1a29-41df-b109-2b8489d4e098" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396") on node "ip-10-0-137-242.ec2.internal" Oct 30 11:34:02.144407 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.144326 4289 reconciler_common.go:299] "Volume detached for volume \"pvc-ae2fc6cc-1a29-41df-b109-2b8489d4e098\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d1e006dc6512396\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"csi-41fb3b285706a9985249f9bffb659d03160e9e3e6288d3728f95a76a8535f583\"" Oct 30 11:34:02.190207 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.190174 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:34:02.190416 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.190395 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="50391e47-44b9-4939-8eec-0e53758e5cae" containerName="otc-container" containerID="cri-o://233ecd11ed973e022ae45e132ee460a7ca0a40cae8ac705fab1220d567e5fc87" gracePeriod=30 Oct 30 11:34:02.279832 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.279795 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:02.433335 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.433311 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:02.453587 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.453560 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:02.547924 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.547894 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50391e47-44b9-4939-8eec-0e53758e5cae-kube-api-access-wsxck\" (UniqueName: \"kubernetes.io/projected/50391e47-44b9-4939-8eec-0e53758e5cae-kube-api-access-wsxck\") pod \"50391e47-44b9-4939-8eec-0e53758e5cae\" (UID: \"50391e47-44b9-4939-8eec-0e53758e5cae\") " Oct 30 11:34:02.548071 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.547950 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50391e47-44b9-4939-8eec-0e53758e5cae-otc-internal\" (UniqueName: \"kubernetes.io/configmap/50391e47-44b9-4939-8eec-0e53758e5cae-otc-internal\") pod \"50391e47-44b9-4939-8eec-0e53758e5cae\" (UID: \"50391e47-44b9-4939-8eec-0e53758e5cae\") " Oct 30 11:34:02.548071 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.548030 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50391e47-44b9-4939-8eec-0e53758e5cae-tls-certs\" (UniqueName: \"kubernetes.io/secret/50391e47-44b9-4939-8eec-0e53758e5cae-tls-certs\") pod \"50391e47-44b9-4939-8eec-0e53758e5cae\" (UID: \"50391e47-44b9-4939-8eec-0e53758e5cae\") " Oct 30 11:34:02.548071 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.548051 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50391e47-44b9-4939-8eec-0e53758e5cae-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/50391e47-44b9-4939-8eec-0e53758e5cae-monitoring-tls-certs\") pod \"50391e47-44b9-4939-8eec-0e53758e5cae\" (UID: \"50391e47-44b9-4939-8eec-0e53758e5cae\") " Oct 30 11:34:02.548216 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.548074 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50391e47-44b9-4939-8eec-0e53758e5cae-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/50391e47-44b9-4939-8eec-0e53758e5cae-data-science-collector-ta-client-cert\") pod \"50391e47-44b9-4939-8eec-0e53758e5cae\" (UID: \"50391e47-44b9-4939-8eec-0e53758e5cae\") " Oct 30 11:34:02.548329 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.548303 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50391e47-44b9-4939-8eec-0e53758e5cae-otc-internal" pod "50391e47-44b9-4939-8eec-0e53758e5cae" (UID: "50391e47-44b9-4939-8eec-0e53758e5cae"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:34:02.550119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.550085 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50391e47-44b9-4939-8eec-0e53758e5cae-kube-api-access-wsxck" pod "50391e47-44b9-4939-8eec-0e53758e5cae" (UID: "50391e47-44b9-4939-8eec-0e53758e5cae"). InnerVolumeSpecName "kube-api-access-wsxck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:34:02.550250 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.550148 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50391e47-44b9-4939-8eec-0e53758e5cae-data-science-collector-ta-client-cert" pod "50391e47-44b9-4939-8eec-0e53758e5cae" (UID: "50391e47-44b9-4939-8eec-0e53758e5cae"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:02.550250 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.550168 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50391e47-44b9-4939-8eec-0e53758e5cae-tls-certs" pod "50391e47-44b9-4939-8eec-0e53758e5cae" (UID: "50391e47-44b9-4939-8eec-0e53758e5cae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:02.550250 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.550235 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50391e47-44b9-4939-8eec-0e53758e5cae-monitoring-tls-certs" pod "50391e47-44b9-4939-8eec-0e53758e5cae" (UID: "50391e47-44b9-4939-8eec-0e53758e5cae"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:02.648909 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.648878 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50391e47-44b9-4939-8eec-0e53758e5cae-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:02.649067 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.648918 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/50391e47-44b9-4939-8eec-0e53758e5cae-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:02.649067 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.648929 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/50391e47-44b9-4939-8eec-0e53758e5cae-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:02.649067 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.648940 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsxck\" (UniqueName: \"kubernetes.io/projected/50391e47-44b9-4939-8eec-0e53758e5cae-kube-api-access-wsxck\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:02.649067 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.648948 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/50391e47-44b9-4939-8eec-0e53758e5cae-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:02.785207 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.785173 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fffff5fd-6828-41bf-88aa-d0e721fedbf0","Type":"ContainerDied","Data":"402542ab3b822777f75520a0c231b0e250a091225f15d4eea92b7aaa52577dfa"} Oct 30 11:34:02.785207 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.785193 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 11:34:02.785452 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.785280 4289 scope.go:122] "RemoveContainer" containerID="65195ead31a8723aa11b20f1e70b1604acb5d2af76798261fc58d4a5f39cfc5e" Oct 30 11:34:02.790669 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.790643 4289 generic.go:356] "Generic (PLEG): container finished" podID="50391e47-44b9-4939-8eec-0e53758e5cae" containerID="233ecd11ed973e022ae45e132ee460a7ca0a40cae8ac705fab1220d567e5fc87" exitCode=0 Oct 30 11:34:02.790789 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.790690 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:02.790789 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.790701 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"50391e47-44b9-4939-8eec-0e53758e5cae","Type":"ContainerDied","Data":"233ecd11ed973e022ae45e132ee460a7ca0a40cae8ac705fab1220d567e5fc87"} Oct 30 11:34:02.790789 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.790762 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"50391e47-44b9-4939-8eec-0e53758e5cae","Type":"ContainerDied","Data":"ed3ef828a2ec71ea529ae530350ca1a0c6cc4cf0db6338f38769c39061160378"} Oct 30 11:34:02.795310 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.795280 4289 scope.go:122] "RemoveContainer" containerID="fafd07d8121393996f64ebec92eeb3b54c2ffe2f52913f42bc9e058b3a5047e6" Oct 30 11:34:02.804247 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.804231 4289 scope.go:122] "RemoveContainer" containerID="b0e5c885df440ef973a805a54acb26e3335937f8893c81ce3589e2afe49b7232" Oct 30 11:34:02.811773 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.811757 4289 scope.go:122] "RemoveContainer" containerID="7961334be6d6a4233aab05762344997aa429a124bbeaf121d6589c5a2f3e0df3" Oct 30 11:34:02.819743 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.819721 4289 scope.go:122] "RemoveContainer" containerID="233ecd11ed973e022ae45e132ee460a7ca0a40cae8ac705fab1220d567e5fc87" Oct 30 11:34:02.826841 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.826825 4289 scope.go:122] "RemoveContainer" containerID="233ecd11ed973e022ae45e132ee460a7ca0a40cae8ac705fab1220d567e5fc87" Oct 30 11:34:02.827100 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:02.827077 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"233ecd11ed973e022ae45e132ee460a7ca0a40cae8ac705fab1220d567e5fc87\": container with ID starting with 233ecd11ed973e022ae45e132ee460a7ca0a40cae8ac705fab1220d567e5fc87 not found: ID does not exist" containerID="233ecd11ed973e022ae45e132ee460a7ca0a40cae8ac705fab1220d567e5fc87" Oct 30 11:34:02.827135 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.827109 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"233ecd11ed973e022ae45e132ee460a7ca0a40cae8ac705fab1220d567e5fc87"} err="failed to get container status \"233ecd11ed973e022ae45e132ee460a7ca0a40cae8ac705fab1220d567e5fc87\": rpc error: code = NotFound desc = could not find container \"233ecd11ed973e022ae45e132ee460a7ca0a40cae8ac705fab1220d567e5fc87\": container with ID starting with 233ecd11ed973e022ae45e132ee460a7ca0a40cae8ac705fab1220d567e5fc87 not found: ID does not exist" Oct 30 11:34:02.840293 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.840267 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 11:34:02.852819 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.852791 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 11:34:02.880175 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.880148 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:34:02.884394 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:02.884369 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:34:03.143615 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:03.143534 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:03.308484 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:03.308420 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:03.624816 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:03.624784 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="098dce60-e115-4404-8e5f-43809c74e6b6" path="/var/lib/kubelet/pods/098dce60-e115-4404-8e5f-43809c74e6b6/volumes" Oct 30 11:34:03.625164 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:03.625152 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17a81880-2dca-4878-bf5a-6f7185f4451e" path="/var/lib/kubelet/pods/17a81880-2dca-4878-bf5a-6f7185f4451e/volumes" Oct 30 11:34:03.625597 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:03.625584 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2835390b-006c-4fc4-81fc-3ab4952f4dc6" path="/var/lib/kubelet/pods/2835390b-006c-4fc4-81fc-3ab4952f4dc6/volumes" Oct 30 11:34:03.626294 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:03.626280 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50391e47-44b9-4939-8eec-0e53758e5cae" path="/var/lib/kubelet/pods/50391e47-44b9-4939-8eec-0e53758e5cae/volumes" Oct 30 11:34:03.626639 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:03.626626 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86e3cacb-a113-4182-b955-fd45c0e77190" path="/var/lib/kubelet/pods/86e3cacb-a113-4182-b955-fd45c0e77190/volumes" Oct 30 11:34:03.627252 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:03.627239 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fffff5fd-6828-41bf-88aa-d0e721fedbf0" path="/var/lib/kubelet/pods/fffff5fd-6828-41bf-88aa-d0e721fedbf0/volumes" Oct 30 11:34:03.635286 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:03.635262 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="4c82f811-e9e7-4c22-ba2b-83e9419752ed" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.117:9093/-/ready\": dial tcp 10.134.0.117:9093: connect: connection refused" Oct 30 11:34:06.333788 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.333748 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:06.537723 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.537684 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:06.601326 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601254 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78"] Oct 30 11:34:06.601601 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601588 4289 cpu_manager.go:414] "containerMap: removing container" podUID="50391e47-44b9-4939-8eec-0e53758e5cae" containerName="otc-container" Oct 30 11:34:06.601653 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601603 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50391e47-44b9-4939-8eec-0e53758e5cae" containerName="otc-container" Oct 30 11:34:06.601653 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601617 4289 cpu_manager.go:414] "containerMap: removing container" podUID="fffff5fd-6828-41bf-88aa-d0e721fedbf0" containerName="thanos-sidecar" Oct 30 11:34:06.601653 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601622 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fffff5fd-6828-41bf-88aa-d0e721fedbf0" containerName="thanos-sidecar" Oct 30 11:34:06.601653 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601632 4289 cpu_manager.go:414] "containerMap: removing container" podUID="17a81880-2dca-4878-bf5a-6f7185f4451e" containerName="method-gate" Oct 30 11:34:06.601653 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601638 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a81880-2dca-4878-bf5a-6f7185f4451e" containerName="method-gate" Oct 30 11:34:06.601653 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601647 4289 cpu_manager.go:414] "containerMap: removing container" podUID="fffff5fd-6828-41bf-88aa-d0e721fedbf0" containerName="prometheus" Oct 30 11:34:06.601653 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601652 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fffff5fd-6828-41bf-88aa-d0e721fedbf0" containerName="prometheus" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601660 4289 cpu_manager.go:414] "containerMap: removing container" podUID="2835390b-006c-4fc4-81fc-3ab4952f4dc6" containerName="kube-rbac-proxy" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601665 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2835390b-006c-4fc4-81fc-3ab4952f4dc6" containerName="kube-rbac-proxy" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601671 4289 cpu_manager.go:414] "containerMap: removing container" podUID="2835390b-006c-4fc4-81fc-3ab4952f4dc6" containerName="prom-label-proxy" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601675 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2835390b-006c-4fc4-81fc-3ab4952f4dc6" containerName="prom-label-proxy" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601681 4289 cpu_manager.go:414] "containerMap: removing container" podUID="fffff5fd-6828-41bf-88aa-d0e721fedbf0" containerName="init-config-reloader" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601686 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fffff5fd-6828-41bf-88aa-d0e721fedbf0" containerName="init-config-reloader" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601691 4289 cpu_manager.go:414] "containerMap: removing container" podUID="fffff5fd-6828-41bf-88aa-d0e721fedbf0" containerName="config-reloader" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601696 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fffff5fd-6828-41bf-88aa-d0e721fedbf0" containerName="config-reloader" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601706 4289 cpu_manager.go:414] "containerMap: removing container" podUID="17a81880-2dca-4878-bf5a-6f7185f4451e" containerName="prom-label-proxy" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601711 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a81880-2dca-4878-bf5a-6f7185f4451e" containerName="prom-label-proxy" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601719 4289 cpu_manager.go:414] "containerMap: removing container" podUID="86e3cacb-a113-4182-b955-fd45c0e77190" containerName="kube-rbac-proxy" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601723 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86e3cacb-a113-4182-b955-fd45c0e77190" containerName="kube-rbac-proxy" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601731 4289 cpu_manager.go:414] "containerMap: removing container" podUID="098dce60-e115-4404-8e5f-43809c74e6b6" containerName="korrel8r" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601736 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="098dce60-e115-4404-8e5f-43809c74e6b6" containerName="korrel8r" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601752 4289 cpu_manager.go:414] "containerMap: removing container" podUID="2835390b-006c-4fc4-81fc-3ab4952f4dc6" containerName="method-gate" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601756 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2835390b-006c-4fc4-81fc-3ab4952f4dc6" containerName="method-gate" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601762 4289 cpu_manager.go:414] "containerMap: removing container" podUID="17a81880-2dca-4878-bf5a-6f7185f4451e" containerName="kube-rbac-proxy" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601767 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a81880-2dca-4878-bf5a-6f7185f4451e" containerName="kube-rbac-proxy" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601811 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="fffff5fd-6828-41bf-88aa-d0e721fedbf0" containerName="prometheus" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601817 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="2835390b-006c-4fc4-81fc-3ab4952f4dc6" containerName="kube-rbac-proxy" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601823 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="2835390b-006c-4fc4-81fc-3ab4952f4dc6" containerName="prom-label-proxy" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601828 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="fffff5fd-6828-41bf-88aa-d0e721fedbf0" containerName="config-reloader" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601835 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a81880-2dca-4878-bf5a-6f7185f4451e" containerName="prom-label-proxy" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601842 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="86e3cacb-a113-4182-b955-fd45c0e77190" containerName="kube-rbac-proxy" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601848 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="098dce60-e115-4404-8e5f-43809c74e6b6" containerName="korrel8r" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601854 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="2835390b-006c-4fc4-81fc-3ab4952f4dc6" containerName="method-gate" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601859 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a81880-2dca-4878-bf5a-6f7185f4451e" containerName="kube-rbac-proxy" Oct 30 11:34:06.601853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601865 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="50391e47-44b9-4939-8eec-0e53758e5cae" containerName="otc-container" Oct 30 11:34:06.602697 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601870 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="fffff5fd-6828-41bf-88aa-d0e721fedbf0" containerName="thanos-sidecar" Oct 30 11:34:06.602697 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.601876 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a81880-2dca-4878-bf5a-6f7185f4451e" containerName="method-gate" Oct 30 11:34:06.606721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.606705 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" Oct 30 11:34:06.627173 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.627146 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 11:34:06.627323 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.627196 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 11:34:06.627400 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.627327 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 11:34:06.627497 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.627416 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-7kwk7\"" Oct 30 11:34:06.628020 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.627999 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 11:34:06.629357 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.629337 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78"] Oct 30 11:34:06.659742 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.659714 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f"] Oct 30 11:34:06.667674 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.667639 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" Oct 30 11:34:06.681702 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.681676 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hdp78\" (UID: \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" Oct 30 11:34:06.681810 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.681710 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-755qz\" (UniqueName: \"kubernetes.io/projected/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-kube-api-access-755qz\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hdp78\" (UID: \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" Oct 30 11:34:06.681810 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.681732 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hdp78\" (UID: \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" Oct 30 11:34:06.681810 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.681799 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hdp78\" (UID: \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" Oct 30 11:34:06.681946 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.681854 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hdp78\" (UID: \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" Oct 30 11:34:06.685395 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.685367 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f"] Oct 30 11:34:06.688733 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.688706 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf"] Oct 30 11:34:06.692231 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.692209 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" Oct 30 11:34:06.715291 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.715267 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf"] Oct 30 11:34:06.726681 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.726651 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:06.727632 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.727610 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 11:34:06.727742 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.727636 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 11:34:06.727742 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.727640 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 11:34:06.728162 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.728143 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-w6p8p\"" Oct 30 11:34:06.728276 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.728181 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 11:34:06.728276 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.728240 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-4v8sb\"" Oct 30 11:34:06.728276 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.728247 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 11:34:06.729184 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.729167 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl"] Oct 30 11:34:06.732688 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.732672 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl" Oct 30 11:34:06.751031 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.751009 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl"] Oct 30 11:34:06.783132 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.783102 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b029b579-df40-408b-a55b-34a9ac9a1f12-tls\") pod \"korrel8r-5cbcc89b8f-wfmrl\" (UID: \"b029b579-df40-408b-a55b-34a9ac9a1f12\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl" Oct 30 11:34:06.783272 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.783136 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34d3af29-0917-4d1d-934d-fbab9aa8f38b-config\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f\" (UID: \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" Oct 30 11:34:06.783272 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.783166 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hdp78\" (UID: \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" Oct 30 11:34:06.783383 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.783348 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hdp78\" (UID: \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" Oct 30 11:34:06.783462 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.783393 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/34d3af29-0917-4d1d-934d-fbab9aa8f38b-client-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f\" (UID: \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" Oct 30 11:34:06.783462 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.783424 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzhtq\" (UniqueName: \"kubernetes.io/projected/34d3af29-0917-4d1d-934d-fbab9aa8f38b-kube-api-access-tzhtq\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f\" (UID: \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" Oct 30 11:34:06.783555 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.783513 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b029b579-df40-408b-a55b-34a9ac9a1f12-config\") pod \"korrel8r-5cbcc89b8f-wfmrl\" (UID: \"b029b579-df40-408b-a55b-34a9ac9a1f12\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl" Oct 30 11:34:06.783555 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:06.783539 4289 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 30 11:34:06.783630 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.783564 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/34d3af29-0917-4d1d-934d-fbab9aa8f38b-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f\" (UID: \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" Oct 30 11:34:06.783630 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:06.783604 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-prometheus-client-cert podName:e79c96e2-ffe5-41d4-823c-b86b416aa4f8 nodeName:}" failed. No retries permitted until 2026-10-30 11:34:07.283583979 +0000 UTC m=+2937.784963446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-hdp78" (UID: "e79c96e2-ffe5-41d4-823c-b86b416aa4f8") : secret "prometheus-operated-tls" not found Oct 30 11:34:06.783732 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.783712 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8cc2\" (UniqueName: \"kubernetes.io/projected/56f0158b-1fcb-4417-8182-995b72813052-kube-api-access-s8cc2\") pod \"data-science-thanos-querier-proxy-7b56895dd5-d6cwf\" (UID: \"56f0158b-1fcb-4417-8182-995b72813052\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" Oct 30 11:34:06.783800 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.783782 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hdp78\" (UID: \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" Oct 30 11:34:06.783854 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.783827 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hdp78\" (UID: \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" Oct 30 11:34:06.783854 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.783845 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/34d3af29-0917-4d1d-934d-fbab9aa8f38b-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f\" (UID: \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" Oct 30 11:34:06.783940 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.783904 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-755qz\" (UniqueName: \"kubernetes.io/projected/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-kube-api-access-755qz\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hdp78\" (UID: \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" Oct 30 11:34:06.783940 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.783933 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56f0158b-1fcb-4417-8182-995b72813052-config\") pod \"data-science-thanos-querier-proxy-7b56895dd5-d6cwf\" (UID: \"56f0158b-1fcb-4417-8182-995b72813052\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" Oct 30 11:34:06.784041 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.783949 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/56f0158b-1fcb-4417-8182-995b72813052-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-d6cwf\" (UID: \"56f0158b-1fcb-4417-8182-995b72813052\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" Oct 30 11:34:06.784041 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.783965 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwgs4\" (UniqueName: \"kubernetes.io/projected/b029b579-df40-408b-a55b-34a9ac9a1f12-kube-api-access-lwgs4\") pod \"korrel8r-5cbcc89b8f-wfmrl\" (UID: \"b029b579-df40-408b-a55b-34a9ac9a1f12\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl" Oct 30 11:34:06.784041 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.784010 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hdp78\" (UID: \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" Oct 30 11:34:06.786096 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.786067 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hdp78\" (UID: \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" Oct 30 11:34:06.786191 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.786123 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hdp78\" (UID: \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" Oct 30 11:34:06.809449 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.809412 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-755qz\" (UniqueName: \"kubernetes.io/projected/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-kube-api-access-755qz\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hdp78\" (UID: \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" Oct 30 11:34:06.827616 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.827588 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 11:34:06.829333 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.829313 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 11:34:06.829464 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.829449 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-gdhrd\"" Oct 30 11:34:06.884880 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.884802 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b029b579-df40-408b-a55b-34a9ac9a1f12-tls\") pod \"korrel8r-5cbcc89b8f-wfmrl\" (UID: \"b029b579-df40-408b-a55b-34a9ac9a1f12\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl" Oct 30 11:34:06.884880 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.884834 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34d3af29-0917-4d1d-934d-fbab9aa8f38b-config\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f\" (UID: \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" Oct 30 11:34:06.885093 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.884931 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/34d3af29-0917-4d1d-934d-fbab9aa8f38b-client-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f\" (UID: \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" Oct 30 11:34:06.885093 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.884958 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzhtq\" (UniqueName: \"kubernetes.io/projected/34d3af29-0917-4d1d-934d-fbab9aa8f38b-kube-api-access-tzhtq\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f\" (UID: \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" Oct 30 11:34:06.885093 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.884978 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b029b579-df40-408b-a55b-34a9ac9a1f12-config\") pod \"korrel8r-5cbcc89b8f-wfmrl\" (UID: \"b029b579-df40-408b-a55b-34a9ac9a1f12\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl" Oct 30 11:34:06.885093 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.885019 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/34d3af29-0917-4d1d-934d-fbab9aa8f38b-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f\" (UID: \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" Oct 30 11:34:06.885093 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.885079 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8cc2\" (UniqueName: \"kubernetes.io/projected/56f0158b-1fcb-4417-8182-995b72813052-kube-api-access-s8cc2\") pod \"data-science-thanos-querier-proxy-7b56895dd5-d6cwf\" (UID: \"56f0158b-1fcb-4417-8182-995b72813052\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" Oct 30 11:34:06.885375 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.885115 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/34d3af29-0917-4d1d-934d-fbab9aa8f38b-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f\" (UID: \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" Oct 30 11:34:06.885375 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.885164 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56f0158b-1fcb-4417-8182-995b72813052-config\") pod \"data-science-thanos-querier-proxy-7b56895dd5-d6cwf\" (UID: \"56f0158b-1fcb-4417-8182-995b72813052\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" Oct 30 11:34:06.885375 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.885187 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/56f0158b-1fcb-4417-8182-995b72813052-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-d6cwf\" (UID: \"56f0158b-1fcb-4417-8182-995b72813052\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" Oct 30 11:34:06.885375 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.885212 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwgs4\" (UniqueName: \"kubernetes.io/projected/b029b579-df40-408b-a55b-34a9ac9a1f12-kube-api-access-lwgs4\") pod \"korrel8r-5cbcc89b8f-wfmrl\" (UID: \"b029b579-df40-408b-a55b-34a9ac9a1f12\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl" Oct 30 11:34:06.885634 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.885517 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/34d3af29-0917-4d1d-934d-fbab9aa8f38b-client-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f\" (UID: \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" Oct 30 11:34:06.885634 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.885580 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34d3af29-0917-4d1d-934d-fbab9aa8f38b-config\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f\" (UID: \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" Oct 30 11:34:06.885924 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.885898 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/34d3af29-0917-4d1d-934d-fbab9aa8f38b-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f\" (UID: \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" Oct 30 11:34:06.886043 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.885956 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b029b579-df40-408b-a55b-34a9ac9a1f12-config\") pod \"korrel8r-5cbcc89b8f-wfmrl\" (UID: \"b029b579-df40-408b-a55b-34a9ac9a1f12\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl" Oct 30 11:34:06.886120 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.886101 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56f0158b-1fcb-4417-8182-995b72813052-config\") pod \"data-science-thanos-querier-proxy-7b56895dd5-d6cwf\" (UID: \"56f0158b-1fcb-4417-8182-995b72813052\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" Oct 30 11:34:06.887319 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.887301 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b029b579-df40-408b-a55b-34a9ac9a1f12-tls\") pod \"korrel8r-5cbcc89b8f-wfmrl\" (UID: \"b029b579-df40-408b-a55b-34a9ac9a1f12\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl" Oct 30 11:34:06.887978 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.887960 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/34d3af29-0917-4d1d-934d-fbab9aa8f38b-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f\" (UID: \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" Oct 30 11:34:06.888114 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.888093 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/56f0158b-1fcb-4417-8182-995b72813052-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-d6cwf\" (UID: \"56f0158b-1fcb-4417-8182-995b72813052\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" Oct 30 11:34:06.890646 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.890622 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:06.908699 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.908670 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzhtq\" (UniqueName: \"kubernetes.io/projected/34d3af29-0917-4d1d-934d-fbab9aa8f38b-kube-api-access-tzhtq\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f\" (UID: \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" Oct 30 11:34:06.908831 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.908811 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwgs4\" (UniqueName: \"kubernetes.io/projected/b029b579-df40-408b-a55b-34a9ac9a1f12-kube-api-access-lwgs4\") pod \"korrel8r-5cbcc89b8f-wfmrl\" (UID: \"b029b579-df40-408b-a55b-34a9ac9a1f12\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl" Oct 30 11:34:06.908886 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.908869 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8cc2\" (UniqueName: \"kubernetes.io/projected/56f0158b-1fcb-4417-8182-995b72813052-kube-api-access-s8cc2\") pod \"data-science-thanos-querier-proxy-7b56895dd5-d6cwf\" (UID: \"56f0158b-1fcb-4417-8182-995b72813052\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" Oct 30 11:34:06.984813 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:06.984780 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" Oct 30 11:34:07.009335 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.009298 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" Oct 30 11:34:07.046053 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.046025 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl" Oct 30 11:34:07.260137 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.260109 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:07.282818 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.282792 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f"] Oct 30 11:34:07.285084 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:34:07.285059 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d3af29_0917_4d1d_934d_fbab9aa8f38b.slice/crio-5a51d27c8d5e70a673c1facafae9f6164f8c4b8d510ae28cb69b3bc43d980dab WatchSource:0}: Error finding container 5a51d27c8d5e70a673c1facafae9f6164f8c4b8d510ae28cb69b3bc43d980dab: Status 404 returned error can't find the container with id 5a51d27c8d5e70a673c1facafae9f6164f8c4b8d510ae28cb69b3bc43d980dab Oct 30 11:34:07.289399 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.289324 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hdp78\" (UID: \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" Oct 30 11:34:07.291644 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.291621 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-hdp78\" (UID: \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" Oct 30 11:34:07.320405 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.320381 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf"] Oct 30 11:34:07.322187 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:34:07.322162 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f0158b_1fcb_4417_8182_995b72813052.slice/crio-1a278c59dfa053513dea07e163ceb4920328733b4c5c89f374409cd44da2cdb4 WatchSource:0}: Error finding container 1a278c59dfa053513dea07e163ceb4920328733b4c5c89f374409cd44da2cdb4: Status 404 returned error can't find the container with id 1a278c59dfa053513dea07e163ceb4920328733b4c5c89f374409cd44da2cdb4 Oct 30 11:34:07.354384 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.354314 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl"] Oct 30 11:34:07.357390 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:34:07.357331 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb029b579_df40_408b_a55b_34a9ac9a1f12.slice/crio-d0d232a6d83274f930e83f901a5700059cf10f34d5b7d24e16c820e386af8c50 WatchSource:0}: Error finding container d0d232a6d83274f930e83f901a5700059cf10f34d5b7d24e16c820e386af8c50: Status 404 returned error can't find the container with id d0d232a6d83274f930e83f901a5700059cf10f34d5b7d24e16c820e386af8c50 Oct 30 11:34:07.433861 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.433825 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:07.519278 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.519216 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" Oct 30 11:34:07.624087 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.624050 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:07.812515 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.812491 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78"] Oct 30 11:34:07.814654 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:34:07.814615 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79c96e2_ffe5_41d4_823c_b86b416aa4f8.slice/crio-6b785a7c878e7f274102c1aeec8a137614cfe2f81c9eb707e82411ad47618296 WatchSource:0}: Error finding container 6b785a7c878e7f274102c1aeec8a137614cfe2f81c9eb707e82411ad47618296: Status 404 returned error can't find the container with id 6b785a7c878e7f274102c1aeec8a137614cfe2f81c9eb707e82411ad47618296 Oct 30 11:34:07.820808 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.820779 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl" event={"ID":"b029b579-df40-408b-a55b-34a9ac9a1f12","Type":"ContainerStarted","Data":"5d148cce2f4c85beee066a557df141368abb906617533a4b36462af5205de3e9"} Oct 30 11:34:07.820948 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.820836 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl" Oct 30 11:34:07.820948 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.820852 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl" event={"ID":"b029b579-df40-408b-a55b-34a9ac9a1f12","Type":"ContainerStarted","Data":"d0d232a6d83274f930e83f901a5700059cf10f34d5b7d24e16c820e386af8c50"} Oct 30 11:34:07.823160 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.823135 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" event={"ID":"56f0158b-1fcb-4417-8182-995b72813052","Type":"ContainerStarted","Data":"3871660646bb1363bcb0074f8a3d0170acd949158caf0d1f266bd062aabd7d4b"} Oct 30 11:34:07.823275 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.823191 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" event={"ID":"56f0158b-1fcb-4417-8182-995b72813052","Type":"ContainerStarted","Data":"e5b7276a8e6bd3c0bbb16389cb5e2a7f00e637c0452ad27b2daca5c5b020d974"} Oct 30 11:34:07.823275 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.823223 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" event={"ID":"56f0158b-1fcb-4417-8182-995b72813052","Type":"ContainerStarted","Data":"63062fde5c2275f5d6ec29953bf56dbd89bd14a059f8532840cc63146c3e564e"} Oct 30 11:34:07.823275 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.823247 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" event={"ID":"56f0158b-1fcb-4417-8182-995b72813052","Type":"ContainerStarted","Data":"1a278c59dfa053513dea07e163ceb4920328733b4c5c89f374409cd44da2cdb4"} Oct 30 11:34:07.825200 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.825180 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" event={"ID":"34d3af29-0917-4d1d-934d-fbab9aa8f38b","Type":"ContainerStarted","Data":"7110f6b5f75631639d64c0ae8b802e0387a9f8ef7ae72fd0119e5cc9d624cf93"} Oct 30 11:34:07.825297 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.825232 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" event={"ID":"34d3af29-0917-4d1d-934d-fbab9aa8f38b","Type":"ContainerStarted","Data":"e8bb2d2ae94770aec5b86d7f8cda3aa22f418a641930f29d747a93d20695bcb9"} Oct 30 11:34:07.825297 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.825273 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" event={"ID":"34d3af29-0917-4d1d-934d-fbab9aa8f38b","Type":"ContainerStarted","Data":"9a36b800cf97aafb3beef4312edd8655cde00df871b74521c11abe998d8d89a6"} Oct 30 11:34:07.825363 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.825310 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" event={"ID":"34d3af29-0917-4d1d-934d-fbab9aa8f38b","Type":"ContainerStarted","Data":"5a51d27c8d5e70a673c1facafae9f6164f8c4b8d510ae28cb69b3bc43d980dab"} Oct 30 11:34:07.861634 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.861575 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl" podStartSLOduration=1.861558528 podStartE2EDuration="1.861558528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:34:07.860721682 +0000 UTC m=+2938.362101168" watchObservedRunningTime="2026-10-30 11:34:07.861558528 +0000 UTC m=+2938.362938013" Oct 30 11:34:07.904262 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.904209 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" podStartSLOduration=1.904194385 podStartE2EDuration="1.904194385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:34:07.902867448 +0000 UTC m=+2938.404246946" watchObservedRunningTime="2026-10-30 11:34:07.904194385 +0000 UTC m=+2938.405573860" Oct 30 11:34:07.944870 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:07.944822 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" podStartSLOduration=1.944807766 podStartE2EDuration="1.944807766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:34:07.944350449 +0000 UTC m=+2938.445729935" watchObservedRunningTime="2026-10-30 11:34:07.944807766 +0000 UTC m=+2938.446187251" Oct 30 11:34:08.282928 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:08.282892 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:08.457463 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:08.457354 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:08.634736 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:08.634698 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="4c82f811-e9e7-4c22-ba2b-83e9419752ed" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.117:9093/-/ready\": dial tcp 10.134.0.117:9093: connect: connection refused" Oct 30 11:34:08.832424 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:08.832388 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" event={"ID":"e79c96e2-ffe5-41d4-823c-b86b416aa4f8","Type":"ContainerStarted","Data":"a80081a532bd69bbbe9d10ea90c89797f6c80f847eef57bf73d5701ddc495782"} Oct 30 11:34:08.832616 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:08.832467 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" event={"ID":"e79c96e2-ffe5-41d4-823c-b86b416aa4f8","Type":"ContainerStarted","Data":"6b785a7c878e7f274102c1aeec8a137614cfe2f81c9eb707e82411ad47618296"} Oct 30 11:34:08.833072 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:08.833053 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" Oct 30 11:34:08.873946 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:08.873898 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" podStartSLOduration=2.873885211 podStartE2EDuration="2.873885211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:34:08.873327232 +0000 UTC m=+2939.374706718" watchObservedRunningTime="2026-10-30 11:34:08.873885211 +0000 UTC m=+2939.375264693" Oct 30 11:34:10.926930 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:10.926894 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:11.168800 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.168765 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:11.341510 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.341464 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:11.551884 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.551849 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:34:11.555794 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.555771 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.569812 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.569786 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:34:11.579155 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.579118 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:11.627311 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.627241 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.627311 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.627308 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f31b5c28-b789-4b2a-a1f6-64d77dffa481-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.627510 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.627378 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f31b5c28-b789-4b2a-a1f6-64d77dffa481-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.627510 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.627464 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.627574 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.627512 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.627574 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.627544 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f31b5c28-b789-4b2a-a1f6-64d77dffa481-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.627637 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.627598 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f31b5c28-b789-4b2a-a1f6-64d77dffa481-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.627637 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.627622 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f31b5c28-b789-4b2a-a1f6-64d77dffa481-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.627698 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.627658 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.627698 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.627688 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.627756 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.627709 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.627756 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.627738 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgfjk\" (UniqueName: \"kubernetes.io/projected/f31b5c28-b789-4b2a-a1f6-64d77dffa481-kube-api-access-mgfjk\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.633764 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.633720 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 11:34:11.633764 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.633732 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 11:34:11.633764 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.633719 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 11:34:11.633764 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.633733 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-jf5dq\"" Oct 30 11:34:11.633764 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.633720 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 11:34:11.633764 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.633770 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 11:34:11.636935 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.636913 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 11:34:11.636935 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.636924 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 11:34:11.637094 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.636970 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 11:34:11.714970 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.714942 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:11.728979 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.728950 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c82f811-e9e7-4c22-ba2b-83e9419752ed-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4c82f811-e9e7-4c22-ba2b-83e9419752ed-alertmanager-data-science-monitoringstack-db\") pod \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " Oct 30 11:34:11.729113 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.728999 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c82f811-e9e7-4c22-ba2b-83e9419752ed-web-config\" (UniqueName: \"kubernetes.io/secret/4c82f811-e9e7-4c22-ba2b-83e9419752ed-web-config\") pod \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " Oct 30 11:34:11.729113 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.729060 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c82f811-e9e7-4c22-ba2b-83e9419752ed-config-volume\" (UniqueName: \"kubernetes.io/secret/4c82f811-e9e7-4c22-ba2b-83e9419752ed-config-volume\") pod \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " Oct 30 11:34:11.729113 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.729083 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c82f811-e9e7-4c22-ba2b-83e9419752ed-config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c82f811-e9e7-4c22-ba2b-83e9419752ed-config-out\") pod \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " Oct 30 11:34:11.729228 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.729136 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c82f811-e9e7-4c22-ba2b-83e9419752ed-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4c82f811-e9e7-4c22-ba2b-83e9419752ed-cluster-tls-config\") pod \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " Oct 30 11:34:11.729283 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.729261 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c82f811-e9e7-4c22-ba2b-83e9419752ed-tls-assets\" (UniqueName: \"kubernetes.io/projected/4c82f811-e9e7-4c22-ba2b-83e9419752ed-tls-assets\") pod \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " Oct 30 11:34:11.729342 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.729310 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c82f811-e9e7-4c22-ba2b-83e9419752ed-alertmanager-data-science-monitoringstack-db" pod "4c82f811-e9e7-4c22-ba2b-83e9419752ed" (UID: "4c82f811-e9e7-4c22-ba2b-83e9419752ed"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:34:11.729342 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.729329 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c82f811-e9e7-4c22-ba2b-83e9419752ed-kube-api-access-b8npm\" (UniqueName: \"kubernetes.io/projected/4c82f811-e9e7-4c22-ba2b-83e9419752ed-kube-api-access-b8npm\") pod \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\" (UID: \"4c82f811-e9e7-4c22-ba2b-83e9419752ed\") " Oct 30 11:34:11.729606 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.729542 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f31b5c28-b789-4b2a-a1f6-64d77dffa481-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.729606 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.729584 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f31b5c28-b789-4b2a-a1f6-64d77dffa481-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.729779 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.729680 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.729779 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.729707 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.729779 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.729759 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f31b5c28-b789-4b2a-a1f6-64d77dffa481-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.729959 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.729786 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f31b5c28-b789-4b2a-a1f6-64d77dffa481-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.729959 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.729820 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f31b5c28-b789-4b2a-a1f6-64d77dffa481-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.729959 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.729849 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.729959 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.729876 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.729959 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.729902 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.729959 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.729926 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgfjk\" (UniqueName: \"kubernetes.io/projected/f31b5c28-b789-4b2a-a1f6-64d77dffa481-kube-api-access-mgfjk\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.730264 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.729995 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.730264 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.730107 4289 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4c82f811-e9e7-4c22-ba2b-83e9419752ed-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:11.731132 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.731101 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f31b5c28-b789-4b2a-a1f6-64d77dffa481-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.731246 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.731189 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f31b5c28-b789-4b2a-a1f6-64d77dffa481-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.731872 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.731834 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f31b5c28-b789-4b2a-a1f6-64d77dffa481-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.733073 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.733045 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c82f811-e9e7-4c22-ba2b-83e9419752ed-config-volume" pod "4c82f811-e9e7-4c22-ba2b-83e9419752ed" (UID: "4c82f811-e9e7-4c22-ba2b-83e9419752ed"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:11.733300 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.733260 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c82f811-e9e7-4c22-ba2b-83e9419752ed-config-out" pod "4c82f811-e9e7-4c22-ba2b-83e9419752ed" (UID: "4c82f811-e9e7-4c22-ba2b-83e9419752ed"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:34:11.733384 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.733345 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c82f811-e9e7-4c22-ba2b-83e9419752ed-kube-api-access-b8npm" pod "4c82f811-e9e7-4c22-ba2b-83e9419752ed" (UID: "4c82f811-e9e7-4c22-ba2b-83e9419752ed"). InnerVolumeSpecName "kube-api-access-b8npm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:34:11.733784 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.733738 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.735826 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.735800 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.736046 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.736014 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.736046 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.736037 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c82f811-e9e7-4c22-ba2b-83e9419752ed-tls-assets" pod "4c82f811-e9e7-4c22-ba2b-83e9419752ed" (UID: "4c82f811-e9e7-4c22-ba2b-83e9419752ed"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:34:11.736501 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.736479 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.737145 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.737105 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f31b5c28-b789-4b2a-a1f6-64d77dffa481-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.737291 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.737268 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.737353 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.737291 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f31b5c28-b789-4b2a-a1f6-64d77dffa481-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.737826 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.737809 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.740124 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.740102 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c82f811-e9e7-4c22-ba2b-83e9419752ed-cluster-tls-config" pod "4c82f811-e9e7-4c22-ba2b-83e9419752ed" (UID: "4c82f811-e9e7-4c22-ba2b-83e9419752ed"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:11.748261 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.748232 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:11.749220 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.749198 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c82f811-e9e7-4c22-ba2b-83e9419752ed-web-config" pod "4c82f811-e9e7-4c22-ba2b-83e9419752ed" (UID: "4c82f811-e9e7-4c22-ba2b-83e9419752ed"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:11.754124 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.754100 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgfjk\" (UniqueName: \"kubernetes.io/projected/f31b5c28-b789-4b2a-a1f6-64d77dffa481-kube-api-access-mgfjk\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:11.830847 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.830814 4289 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4c82f811-e9e7-4c22-ba2b-83e9419752ed-cluster-tls-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:11.830847 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.830842 4289 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c82f811-e9e7-4c22-ba2b-83e9419752ed-tls-assets\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:11.830847 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.830853 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8npm\" (UniqueName: \"kubernetes.io/projected/4c82f811-e9e7-4c22-ba2b-83e9419752ed-kube-api-access-b8npm\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:11.831086 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.830862 4289 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c82f811-e9e7-4c22-ba2b-83e9419752ed-web-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:11.831086 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.830870 4289 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4c82f811-e9e7-4c22-ba2b-83e9419752ed-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:11.831086 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.830877 4289 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c82f811-e9e7-4c22-ba2b-83e9419752ed-config-out\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:11.850313 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.850280 4289 generic.go:356] "Generic (PLEG): container finished" podID="4c82f811-e9e7-4c22-ba2b-83e9419752ed" containerID="81761ef6c91ecf83242e65ed237a1c6747f899843ace3fd6af253290b8d204c3" exitCode=0 Oct 30 11:34:11.850521 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.850345 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:11.850521 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.850361 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"4c82f811-e9e7-4c22-ba2b-83e9419752ed","Type":"ContainerDied","Data":"81761ef6c91ecf83242e65ed237a1c6747f899843ace3fd6af253290b8d204c3"} Oct 30 11:34:11.850521 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.850451 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"4c82f811-e9e7-4c22-ba2b-83e9419752ed","Type":"ContainerDied","Data":"148bc5fe0e663903e41bd57957e5de2e8d19310462c1983dfe1d8dfb27c50bc4"} Oct 30 11:34:11.850521 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.850493 4289 scope.go:122] "RemoveContainer" containerID="fab688ee088d845b5a234e36b02d38d08d033d600d280016f6e5ac74b995905b" Oct 30 11:34:11.859539 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.859523 4289 scope.go:122] "RemoveContainer" containerID="81761ef6c91ecf83242e65ed237a1c6747f899843ace3fd6af253290b8d204c3" Oct 30 11:34:11.867329 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.867310 4289 scope.go:122] "RemoveContainer" containerID="1d07d9edfe5f4982678252182b29f91cb2b57ebf3a51268711c8470db2811f53" Oct 30 11:34:11.877345 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.876951 4289 scope.go:122] "RemoveContainer" containerID="fab688ee088d845b5a234e36b02d38d08d033d600d280016f6e5ac74b995905b" Oct 30 11:34:11.877619 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:11.877366 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab688ee088d845b5a234e36b02d38d08d033d600d280016f6e5ac74b995905b\": container with ID starting with fab688ee088d845b5a234e36b02d38d08d033d600d280016f6e5ac74b995905b not found: ID does not exist" containerID="fab688ee088d845b5a234e36b02d38d08d033d600d280016f6e5ac74b995905b" Oct 30 11:34:11.877619 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.877404 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab688ee088d845b5a234e36b02d38d08d033d600d280016f6e5ac74b995905b"} err="failed to get container status \"fab688ee088d845b5a234e36b02d38d08d033d600d280016f6e5ac74b995905b\": rpc error: code = NotFound desc = could not find container \"fab688ee088d845b5a234e36b02d38d08d033d600d280016f6e5ac74b995905b\": container with ID starting with fab688ee088d845b5a234e36b02d38d08d033d600d280016f6e5ac74b995905b not found: ID does not exist" Oct 30 11:34:11.877619 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.877456 4289 scope.go:122] "RemoveContainer" containerID="81761ef6c91ecf83242e65ed237a1c6747f899843ace3fd6af253290b8d204c3" Oct 30 11:34:11.877829 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:11.877761 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81761ef6c91ecf83242e65ed237a1c6747f899843ace3fd6af253290b8d204c3\": container with ID starting with 81761ef6c91ecf83242e65ed237a1c6747f899843ace3fd6af253290b8d204c3 not found: ID does not exist" containerID="81761ef6c91ecf83242e65ed237a1c6747f899843ace3fd6af253290b8d204c3" Oct 30 11:34:11.877829 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.877794 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81761ef6c91ecf83242e65ed237a1c6747f899843ace3fd6af253290b8d204c3"} err="failed to get container status \"81761ef6c91ecf83242e65ed237a1c6747f899843ace3fd6af253290b8d204c3\": rpc error: code = NotFound desc = could not find container \"81761ef6c91ecf83242e65ed237a1c6747f899843ace3fd6af253290b8d204c3\": container with ID starting with 81761ef6c91ecf83242e65ed237a1c6747f899843ace3fd6af253290b8d204c3 not found: ID does not exist" Oct 30 11:34:11.877829 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.877817 4289 scope.go:122] "RemoveContainer" containerID="1d07d9edfe5f4982678252182b29f91cb2b57ebf3a51268711c8470db2811f53" Oct 30 11:34:11.880178 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:11.880155 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d07d9edfe5f4982678252182b29f91cb2b57ebf3a51268711c8470db2811f53\": container with ID starting with 1d07d9edfe5f4982678252182b29f91cb2b57ebf3a51268711c8470db2811f53 not found: ID does not exist" containerID="1d07d9edfe5f4982678252182b29f91cb2b57ebf3a51268711c8470db2811f53" Oct 30 11:34:11.880276 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.880185 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d07d9edfe5f4982678252182b29f91cb2b57ebf3a51268711c8470db2811f53"} err="failed to get container status \"1d07d9edfe5f4982678252182b29f91cb2b57ebf3a51268711c8470db2811f53\": rpc error: code = NotFound desc = could not find container \"1d07d9edfe5f4982678252182b29f91cb2b57ebf3a51268711c8470db2811f53\": container with ID starting with 1d07d9edfe5f4982678252182b29f91cb2b57ebf3a51268711c8470db2811f53 not found: ID does not exist" Oct 30 11:34:11.905665 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.905632 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:34:11.913359 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:11.913326 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:34:13.591290 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.591256 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:34:13.591900 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.591878 4289 cpu_manager.go:414] "containerMap: removing container" podUID="4c82f811-e9e7-4c22-ba2b-83e9419752ed" containerName="init-config-reloader" Oct 30 11:34:13.591984 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.591901 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c82f811-e9e7-4c22-ba2b-83e9419752ed" containerName="init-config-reloader" Oct 30 11:34:13.591984 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.591916 4289 cpu_manager.go:414] "containerMap: removing container" podUID="4c82f811-e9e7-4c22-ba2b-83e9419752ed" containerName="alertmanager" Oct 30 11:34:13.591984 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.591924 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c82f811-e9e7-4c22-ba2b-83e9419752ed" containerName="alertmanager" Oct 30 11:34:13.591984 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.591950 4289 cpu_manager.go:414] "containerMap: removing container" podUID="4c82f811-e9e7-4c22-ba2b-83e9419752ed" containerName="config-reloader" Oct 30 11:34:13.591984 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.591957 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c82f811-e9e7-4c22-ba2b-83e9419752ed" containerName="config-reloader" Oct 30 11:34:13.592240 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.592049 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c82f811-e9e7-4c22-ba2b-83e9419752ed" containerName="alertmanager" Oct 30 11:34:13.592240 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.592064 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c82f811-e9e7-4c22-ba2b-83e9419752ed" containerName="config-reloader" Oct 30 11:34:13.597287 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.597262 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:13.629049 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.629020 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c82f811-e9e7-4c22-ba2b-83e9419752ed" path="/var/lib/kubelet/pods/4c82f811-e9e7-4c22-ba2b-83e9419752ed/volumes" Oct 30 11:34:13.629376 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.629362 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:34:13.646034 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.646004 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c34bf4cd-d232-45d9-8e12-40ea4859b874-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:13.646222 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.646053 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c34bf4cd-d232-45d9-8e12-40ea4859b874-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:13.646222 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.646128 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c34bf4cd-d232-45d9-8e12-40ea4859b874-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:13.646344 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.646252 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c34bf4cd-d232-45d9-8e12-40ea4859b874-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:13.646344 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.646298 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgjvb\" (UniqueName: \"kubernetes.io/projected/c34bf4cd-d232-45d9-8e12-40ea4859b874-kube-api-access-hgjvb\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:13.646448 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.646404 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c34bf4cd-d232-45d9-8e12-40ea4859b874-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:13.646494 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.646448 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c34bf4cd-d232-45d9-8e12-40ea4859b874-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:13.683065 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.683041 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 11:34:13.683065 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.683054 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 11:34:13.683065 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.683072 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 11:34:13.683266 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.683103 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 11:34:13.683266 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.683136 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-ltmbg\"" Oct 30 11:34:13.747806 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.747775 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c34bf4cd-d232-45d9-8e12-40ea4859b874-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:13.747806 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.747807 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c34bf4cd-d232-45d9-8e12-40ea4859b874-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:13.748001 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.747858 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c34bf4cd-d232-45d9-8e12-40ea4859b874-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:13.748001 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.747894 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgjvb\" (UniqueName: \"kubernetes.io/projected/c34bf4cd-d232-45d9-8e12-40ea4859b874-kube-api-access-hgjvb\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:13.748001 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.747948 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c34bf4cd-d232-45d9-8e12-40ea4859b874-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:13.748107 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.748073 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c34bf4cd-d232-45d9-8e12-40ea4859b874-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:13.748152 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.748136 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c34bf4cd-d232-45d9-8e12-40ea4859b874-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:13.748319 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.748279 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c34bf4cd-d232-45d9-8e12-40ea4859b874-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:13.750257 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.750230 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c34bf4cd-d232-45d9-8e12-40ea4859b874-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:13.750410 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.750390 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c34bf4cd-d232-45d9-8e12-40ea4859b874-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:13.750478 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.750454 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c34bf4cd-d232-45d9-8e12-40ea4859b874-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:13.750670 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.750652 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c34bf4cd-d232-45d9-8e12-40ea4859b874-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:13.750715 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.750683 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c34bf4cd-d232-45d9-8e12-40ea4859b874-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:13.771348 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.771326 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgjvb\" (UniqueName: \"kubernetes.io/projected/c34bf4cd-d232-45d9-8e12-40ea4859b874-kube-api-access-hgjvb\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:13.914404 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:13.914319 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:14.246758 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:14.246733 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:34:14.247585 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:34:14.247558 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34bf4cd_d232_45d9_8e12_40ea4859b874.slice/crio-0bf331c51f4a1ed2514ea18ef6482c39e75ec8a667df17fb7923b3caf5aeb183 WatchSource:0}: Error finding container 0bf331c51f4a1ed2514ea18ef6482c39e75ec8a667df17fb7923b3caf5aeb183: Status 404 returned error can't find the container with id 0bf331c51f4a1ed2514ea18ef6482c39e75ec8a667df17fb7923b3caf5aeb183 Oct 30 11:34:14.872794 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:14.872703 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c34bf4cd-d232-45d9-8e12-40ea4859b874","Type":"ContainerStarted","Data":"d22a0e114d493b8afe9fc08f4ec40c048a1c1bffcfadb06f660baa005269d04d"} Oct 30 11:34:14.872794 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:14.872757 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c34bf4cd-d232-45d9-8e12-40ea4859b874","Type":"ContainerStarted","Data":"0bf331c51f4a1ed2514ea18ef6482c39e75ec8a667df17fb7923b3caf5aeb183"} Oct 30 11:34:15.423685 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:15.423652 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:15.846045 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:15.846012 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" Oct 30 11:34:16.374991 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:16.374955 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:16.403969 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:16.403939 4289 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:16.476506 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:16.476470 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:16.476677 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:16.476620 4289 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:16.491239 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:16.491211 4289 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") DevicePath \"csi-ea06c66241a0a003a5daff5d9fc602f40e53768b00b104c5e4118b92d558c090\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:16.593651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:16.593619 4289 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7342da0537aa528cc1545d2018ac3fd0382dfe4f670067e58cf8ba34bb891174/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:16.620498 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:16.620469 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:16.681321 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:16.681245 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:17.010920 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:17.010894 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:34:17.012772 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:34:17.012743 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31b5c28_b789_4b2a_a1f6_64d77dffa481.slice/crio-a87a0754589140645a6962d914b751d2af594708537ea6c9ce011876a185e28c WatchSource:0}: Error finding container a87a0754589140645a6962d914b751d2af594708537ea6c9ce011876a185e28c: Status 404 returned error can't find the container with id a87a0754589140645a6962d914b751d2af594708537ea6c9ce011876a185e28c Oct 30 11:34:17.886642 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:17.886607 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f31b5c28-b789-4b2a-a1f6-64d77dffa481","Type":"ContainerStarted","Data":"cc4f83084404b18077aa2eff2447531865ba015c7fd17c0c77f2310f43fdbc6d"} Oct 30 11:34:17.887083 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:17.886679 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f31b5c28-b789-4b2a-a1f6-64d77dffa481","Type":"ContainerStarted","Data":"a87a0754589140645a6962d914b751d2af594708537ea6c9ce011876a185e28c"} Oct 30 11:34:18.834975 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:18.834940 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl" Oct 30 11:34:18.894221 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:18.894185 4289 generic.go:356] "Generic (PLEG): container finished" podID="c34bf4cd-d232-45d9-8e12-40ea4859b874" containerID="d22a0e114d493b8afe9fc08f4ec40c048a1c1bffcfadb06f660baa005269d04d" exitCode=0 Oct 30 11:34:18.894747 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:18.894250 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c34bf4cd-d232-45d9-8e12-40ea4859b874","Type":"ContainerDied","Data":"d22a0e114d493b8afe9fc08f4ec40c048a1c1bffcfadb06f660baa005269d04d"} Oct 30 11:34:19.904479 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:19.904370 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c34bf4cd-d232-45d9-8e12-40ea4859b874","Type":"ContainerStarted","Data":"27c0870e6bd9d5d9833b9129e75087f67efaaf7346eb5eb11559b8e43023464e"} Oct 30 11:34:19.904479 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:19.904459 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c34bf4cd-d232-45d9-8e12-40ea4859b874","Type":"ContainerStarted","Data":"198f4fd92c27383cfbeeb2e5931a7ccbe9d492e2ad563c6e7486be5eda12b260"} Oct 30 11:34:19.904991 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:19.904686 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:19.976276 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:19.976225 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.389793138 podStartE2EDuration="6.97621127s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.586418132s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:34:19.963488288 +0000 UTC m=+2950.464867771" watchObservedRunningTime="2026-10-30 11:34:19.97621127 +0000 UTC m=+2950.477590790" Oct 30 11:34:21.916544 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:21.916509 4289 generic.go:356] "Generic (PLEG): container finished" podID="f31b5c28-b789-4b2a-a1f6-64d77dffa481" containerID="cc4f83084404b18077aa2eff2447531865ba015c7fd17c0c77f2310f43fdbc6d" exitCode=0 Oct 30 11:34:21.917052 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:21.916592 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f31b5c28-b789-4b2a-a1f6-64d77dffa481","Type":"ContainerDied","Data":"cc4f83084404b18077aa2eff2447531865ba015c7fd17c0c77f2310f43fdbc6d"} Oct 30 11:34:22.924936 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:22.924895 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f31b5c28-b789-4b2a-a1f6-64d77dffa481","Type":"ContainerStarted","Data":"78fa504f2cea0ded06c3699d3a6cb64372eb90e1f27d7fd1f324ee9ca1db8ec1"} Oct 30 11:34:22.925473 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:22.924964 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f31b5c28-b789-4b2a-a1f6-64d77dffa481","Type":"ContainerStarted","Data":"df76779925e290ac771ceba64e602d0dc83eb576effab05870852f41c741d61e"} Oct 30 11:34:22.925473 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:22.924989 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f31b5c28-b789-4b2a-a1f6-64d77dffa481","Type":"ContainerStarted","Data":"36ec3031da98de911ccd0c0b74981aeeb4673cc0af6c46cdc4367d874530ce8f"} Oct 30 11:34:23.002862 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:23.002802 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=8.216434064 podStartE2EDuration="12.002788255s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.786354191s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:34:22.988967396 +0000 UTC m=+2953.490346881" watchObservedRunningTime="2026-10-30 11:34:23.002788255 +0000 UTC m=+2953.504167739" Oct 30 11:34:24.662290 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:24.662251 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:24.835451 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:24.835396 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:25.695636 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:25.695601 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:25.910969 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:25.910942 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:25.941789 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:25.941754 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:26.141876 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:26.141840 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:26.258190 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:26.258154 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:34:26.681498 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:26.681468 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:27.457206 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:27.457166 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:27.631308 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:27.631254 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:27.913521 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:27.913487 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:34:27.917271 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:27.917251 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:27.944334 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:27.944302 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-phltc\"" Oct 30 11:34:27.944507 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:27.944383 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 11:34:27.947036 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:27.947004 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:34:27.986305 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:27.986262 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:27.986476 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:27.986323 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:27.986476 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:27.986415 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:27.986563 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:27.986495 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:27.986563 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:27.986513 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qfjz\" (UniqueName: \"kubernetes.io/projected/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-kube-api-access-4qfjz\") pod \"data-science-collector-collector-1\" (UID: \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:27.999756 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:27.999732 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-e4cadd38\"" Oct 30 11:34:28.032292 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:28.032269 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 11:34:28.032450 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:28.032312 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 11:34:28.087330 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:28.087302 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:28.087522 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:28.087353 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:28.087522 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:28.087393 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:28.087522 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:28.087450 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:28.087522 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:28.087488 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qfjz\" (UniqueName: \"kubernetes.io/projected/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-kube-api-access-4qfjz\") pod \"data-science-collector-collector-1\" (UID: \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:28.088155 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:28.088134 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:28.089909 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:28.089889 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:28.090004 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:28.089964 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:28.090067 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:28.090023 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:28.110677 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:28.110649 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qfjz\" (UniqueName: \"kubernetes.io/projected/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-kube-api-access-4qfjz\") pod \"data-science-collector-collector-1\" (UID: \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:28.229558 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:28.229531 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:28.527731 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:28.527703 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:34:28.529234 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:34:28.529208 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda26be20c_bd2b_4b59_83c3_cf3b13f0d159.slice/crio-df4224ea1bcdb101853fd93e440a6ce3cb3e624103413fe8ce7fcb840b9d7887 WatchSource:0}: Error finding container df4224ea1bcdb101853fd93e440a6ce3cb3e624103413fe8ce7fcb840b9d7887: Status 404 returned error can't find the container with id df4224ea1bcdb101853fd93e440a6ce3cb3e624103413fe8ce7fcb840b9d7887 Oct 30 11:34:28.531592 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:28.531558 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:34:28.531681 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:28.531616 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:34:28.954035 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:28.953949 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a26be20c-bd2b-4b59-83c3-cf3b13f0d159","Type":"ContainerStarted","Data":"011d8915402977e51b97bb392541cb072ea7969990cbc1b09025b2b905ddb433"} Oct 30 11:34:28.954035 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:28.954003 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a26be20c-bd2b-4b59-83c3-cf3b13f0d159","Type":"ContainerStarted","Data":"df4224ea1bcdb101853fd93e440a6ce3cb3e624103413fe8ce7fcb840b9d7887"} Oct 30 11:34:29.005112 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:29.005061 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.005048723 podStartE2EDuration="2.005048723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:34:29.00387816 +0000 UTC m=+2959.505257647" watchObservedRunningTime="2026-10-30 11:34:29.005048723 +0000 UTC m=+2959.506428211" Oct 30 11:34:31.682092 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:31.682048 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:31.687531 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:31.687509 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:31.972885 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:31.972854 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:37.597656 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:37.597612 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:37.762729 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:37.762693 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:39.162666 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:39.162633 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:34:39.163088 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:39.162818 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="a26be20c-bd2b-4b59-83c3-cf3b13f0d159" containerName="otc-container" containerID="cri-o://011d8915402977e51b97bb392541cb072ea7969990cbc1b09025b2b905ddb433" gracePeriod=30 Oct 30 11:34:39.346310 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:39.346278 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:39.413216 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:39.413156 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:39.489767 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:39.489730 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-tls-certs\" (UniqueName: \"kubernetes.io/secret/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-tls-certs\") pod \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\" (UID: \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\") " Oct 30 11:34:39.489959 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:39.489773 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-data-science-collector-ta-client-cert\") pod \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\" (UID: \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\") " Oct 30 11:34:39.489959 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:39.489838 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-otc-internal\" (UniqueName: \"kubernetes.io/configmap/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-otc-internal\") pod \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\" (UID: \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\") " Oct 30 11:34:39.489959 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:39.489942 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-monitoring-tls-certs\") pod \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\" (UID: \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\") " Oct 30 11:34:39.490117 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:39.489981 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-kube-api-access-4qfjz\" (UniqueName: \"kubernetes.io/projected/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-kube-api-access-4qfjz\") pod \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\" (UID: \"a26be20c-bd2b-4b59-83c3-cf3b13f0d159\") " Oct 30 11:34:39.490286 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:39.490258 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-otc-internal" pod "a26be20c-bd2b-4b59-83c3-cf3b13f0d159" (UID: "a26be20c-bd2b-4b59-83c3-cf3b13f0d159"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:34:39.492126 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:39.492096 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-tls-certs" pod "a26be20c-bd2b-4b59-83c3-cf3b13f0d159" (UID: "a26be20c-bd2b-4b59-83c3-cf3b13f0d159"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:39.492126 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:39.492122 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-data-science-collector-ta-client-cert" pod "a26be20c-bd2b-4b59-83c3-cf3b13f0d159" (UID: "a26be20c-bd2b-4b59-83c3-cf3b13f0d159"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:39.492305 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:39.492254 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-kube-api-access-4qfjz" pod "a26be20c-bd2b-4b59-83c3-cf3b13f0d159" (UID: "a26be20c-bd2b-4b59-83c3-cf3b13f0d159"). InnerVolumeSpecName "kube-api-access-4qfjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:34:39.492305 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:39.492265 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-monitoring-tls-certs" pod "a26be20c-bd2b-4b59-83c3-cf3b13f0d159" (UID: "a26be20c-bd2b-4b59-83c3-cf3b13f0d159"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:39.512887 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:39.512858 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:39.591320 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:39.591288 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:39.591320 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:39.591314 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qfjz\" (UniqueName: \"kubernetes.io/projected/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-kube-api-access-4qfjz\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:39.591320 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:39.591325 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:39.591577 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:39.591337 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:39.591577 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:39.591346 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a26be20c-bd2b-4b59-83c3-cf3b13f0d159-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:40.004851 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.004820 4289 generic.go:356] "Generic (PLEG): container finished" podID="a26be20c-bd2b-4b59-83c3-cf3b13f0d159" containerID="011d8915402977e51b97bb392541cb072ea7969990cbc1b09025b2b905ddb433" exitCode=0 Oct 30 11:34:40.005059 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.004883 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:40.005059 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.004869 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a26be20c-bd2b-4b59-83c3-cf3b13f0d159","Type":"ContainerDied","Data":"011d8915402977e51b97bb392541cb072ea7969990cbc1b09025b2b905ddb433"} Oct 30 11:34:40.005059 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.005012 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a26be20c-bd2b-4b59-83c3-cf3b13f0d159","Type":"ContainerDied","Data":"df4224ea1bcdb101853fd93e440a6ce3cb3e624103413fe8ce7fcb840b9d7887"} Oct 30 11:34:40.005059 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.005049 4289 scope.go:122] "RemoveContainer" containerID="011d8915402977e51b97bb392541cb072ea7969990cbc1b09025b2b905ddb433" Oct 30 11:34:40.013795 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.013764 4289 scope.go:122] "RemoveContainer" containerID="011d8915402977e51b97bb392541cb072ea7969990cbc1b09025b2b905ddb433" Oct 30 11:34:40.014066 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:40.014037 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"011d8915402977e51b97bb392541cb072ea7969990cbc1b09025b2b905ddb433\": container with ID starting with 011d8915402977e51b97bb392541cb072ea7969990cbc1b09025b2b905ddb433 not found: ID does not exist" containerID="011d8915402977e51b97bb392541cb072ea7969990cbc1b09025b2b905ddb433" Oct 30 11:34:40.014164 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.014077 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"011d8915402977e51b97bb392541cb072ea7969990cbc1b09025b2b905ddb433"} err="failed to get container status \"011d8915402977e51b97bb392541cb072ea7969990cbc1b09025b2b905ddb433\": rpc error: code = NotFound desc = could not find container \"011d8915402977e51b97bb392541cb072ea7969990cbc1b09025b2b905ddb433\": container with ID starting with 011d8915402977e51b97bb392541cb072ea7969990cbc1b09025b2b905ddb433 not found: ID does not exist" Oct 30 11:34:40.055697 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.055662 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:34:40.067779 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.067750 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:34:40.095876 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.095846 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:34:40.096485 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.096464 4289 cpu_manager.go:414] "containerMap: removing container" podUID="a26be20c-bd2b-4b59-83c3-cf3b13f0d159" containerName="otc-container" Oct 30 11:34:40.096556 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.096489 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a26be20c-bd2b-4b59-83c3-cf3b13f0d159" containerName="otc-container" Oct 30 11:34:40.096644 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.096631 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="a26be20c-bd2b-4b59-83c3-cf3b13f0d159" containerName="otc-container" Oct 30 11:34:40.101088 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.101072 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:40.118480 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.118457 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-e4cadd38\"" Oct 30 11:34:40.120140 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.120113 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 11:34:40.120140 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.120114 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 11:34:40.120671 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.120652 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:34:40.130659 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.130641 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 11:34:40.130737 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.130668 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-phltc\"" Oct 30 11:34:40.196236 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.196203 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1808eff5-6fef-47ae-b389-2f8e0c4859c4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:40.196236 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.196240 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1808eff5-6fef-47ae-b389-2f8e0c4859c4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:40.196654 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.196264 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4sx5\" (UniqueName: \"kubernetes.io/projected/1808eff5-6fef-47ae-b389-2f8e0c4859c4-kube-api-access-m4sx5\") pod \"data-science-collector-collector-1\" (UID: \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:40.196654 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.196361 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1808eff5-6fef-47ae-b389-2f8e0c4859c4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:40.196654 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.196384 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1808eff5-6fef-47ae-b389-2f8e0c4859c4-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:40.297188 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.297107 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1808eff5-6fef-47ae-b389-2f8e0c4859c4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:40.297188 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.297145 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1808eff5-6fef-47ae-b389-2f8e0c4859c4-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:40.297424 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.297195 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1808eff5-6fef-47ae-b389-2f8e0c4859c4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:40.297424 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.297214 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1808eff5-6fef-47ae-b389-2f8e0c4859c4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:40.297424 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.297236 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4sx5\" (UniqueName: \"kubernetes.io/projected/1808eff5-6fef-47ae-b389-2f8e0c4859c4-kube-api-access-m4sx5\") pod \"data-science-collector-collector-1\" (UID: \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:40.297904 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.297876 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1808eff5-6fef-47ae-b389-2f8e0c4859c4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:40.299725 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.299704 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1808eff5-6fef-47ae-b389-2f8e0c4859c4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:40.299820 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.299758 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1808eff5-6fef-47ae-b389-2f8e0c4859c4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:40.299820 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.299792 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1808eff5-6fef-47ae-b389-2f8e0c4859c4-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:40.321926 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.321901 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4sx5\" (UniqueName: \"kubernetes.io/projected/1808eff5-6fef-47ae-b389-2f8e0c4859c4-kube-api-access-m4sx5\") pod \"data-science-collector-collector-1\" (UID: \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:40.413858 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.413824 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:40.713989 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.713965 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:34:40.716256 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:34:40.716227 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1808eff5_6fef_47ae_b389_2f8e0c4859c4.slice/crio-84f07e023a3077521a3635ed8817bcca6df1f52e0158658d39d57102e99e7621 WatchSource:0}: Error finding container 84f07e023a3077521a3635ed8817bcca6df1f52e0158658d39d57102e99e7621: Status 404 returned error can't find the container with id 84f07e023a3077521a3635ed8817bcca6df1f52e0158658d39d57102e99e7621 Oct 30 11:34:40.718919 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.718885 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:34:40.719037 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:40.718959 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:34:41.011777 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:41.011746 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1808eff5-6fef-47ae-b389-2f8e0c4859c4","Type":"ContainerStarted","Data":"9f0fee691b66ea94933da307d7d13137fe95e3ba454f734a443d765491fb3675"} Oct 30 11:34:41.011965 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:41.011796 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1808eff5-6fef-47ae-b389-2f8e0c4859c4","Type":"ContainerStarted","Data":"84f07e023a3077521a3635ed8817bcca6df1f52e0158658d39d57102e99e7621"} Oct 30 11:34:41.054928 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:41.054873 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.054857253 podStartE2EDuration="1.054857253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:34:41.054024953 +0000 UTC m=+2971.555404437" watchObservedRunningTime="2026-10-30 11:34:41.054857253 +0000 UTC m=+2971.556236738" Oct 30 11:34:41.624671 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:41.624637 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a26be20c-bd2b-4b59-83c3-cf3b13f0d159" path="/var/lib/kubelet/pods/a26be20c-bd2b-4b59-83c3-cf3b13f0d159/volumes" Oct 30 11:34:43.269167 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.269133 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78"] Oct 30 11:34:43.269673 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.269337 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" podUID="e79c96e2-ffe5-41d4-823c-b86b416aa4f8" containerName="kube-rbac-proxy" containerID="cri-o://a80081a532bd69bbbe9d10ea90c89797f6c80f847eef57bf73d5701ddc495782" gracePeriod=30 Oct 30 11:34:43.281216 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.281189 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f"] Oct 30 11:34:43.281450 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.281410 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" podUID="34d3af29-0917-4d1d-934d-fbab9aa8f38b" containerName="kube-rbac-proxy" containerID="cri-o://9a36b800cf97aafb3beef4312edd8655cde00df871b74521c11abe998d8d89a6" gracePeriod=30 Oct 30 11:34:43.281511 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.281420 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" podUID="34d3af29-0917-4d1d-934d-fbab9aa8f38b" containerName="prom-label-proxy" containerID="cri-o://7110f6b5f75631639d64c0ae8b802e0387a9f8ef7ae72fd0119e5cc9d624cf93" gracePeriod=30 Oct 30 11:34:43.281511 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.281455 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" podUID="34d3af29-0917-4d1d-934d-fbab9aa8f38b" containerName="method-gate" containerID="cri-o://e8bb2d2ae94770aec5b86d7f8cda3aa22f418a641930f29d747a93d20695bcb9" gracePeriod=30 Oct 30 11:34:43.296748 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.296722 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf"] Oct 30 11:34:43.296952 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.296933 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" podUID="56f0158b-1fcb-4417-8182-995b72813052" containerName="kube-rbac-proxy" containerID="cri-o://63062fde5c2275f5d6ec29953bf56dbd89bd14a059f8532840cc63146c3e564e" gracePeriod=30 Oct 30 11:34:43.297054 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.296959 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" podUID="56f0158b-1fcb-4417-8182-995b72813052" containerName="method-gate" containerID="cri-o://e5b7276a8e6bd3c0bbb16389cb5e2a7f00e637c0452ad27b2daca5c5b020d974" gracePeriod=30 Oct 30 11:34:43.297054 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.296946 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" podUID="56f0158b-1fcb-4417-8182-995b72813052" containerName="prom-label-proxy" containerID="cri-o://3871660646bb1363bcb0074f8a3d0170acd949158caf0d1f266bd062aabd7d4b" gracePeriod=30 Oct 30 11:34:43.312355 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.312327 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl"] Oct 30 11:34:43.312633 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.312591 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl" podUID="b029b579-df40-408b-a55b-34a9ac9a1f12" containerName="korrel8r" containerID="cri-o://5d148cce2f4c85beee066a557df141368abb906617533a4b36462af5205de3e9" gracePeriod=30 Oct 30 11:34:43.588697 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.588661 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" Oct 30 11:34:43.628517 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.628489 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56f0158b-1fcb-4417-8182-995b72813052-tls\" (UniqueName: \"kubernetes.io/secret/56f0158b-1fcb-4417-8182-995b72813052-tls\") pod \"56f0158b-1fcb-4417-8182-995b72813052\" (UID: \"56f0158b-1fcb-4417-8182-995b72813052\") " Oct 30 11:34:43.628865 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.628836 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56f0158b-1fcb-4417-8182-995b72813052-config\" (UniqueName: \"kubernetes.io/configmap/56f0158b-1fcb-4417-8182-995b72813052-config\") pod \"56f0158b-1fcb-4417-8182-995b72813052\" (UID: \"56f0158b-1fcb-4417-8182-995b72813052\") " Oct 30 11:34:43.628970 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.628912 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f0158b-1fcb-4417-8182-995b72813052-kube-api-access-s8cc2\" (UniqueName: \"kubernetes.io/projected/56f0158b-1fcb-4417-8182-995b72813052-kube-api-access-s8cc2\") pod \"56f0158b-1fcb-4417-8182-995b72813052\" (UID: \"56f0158b-1fcb-4417-8182-995b72813052\") " Oct 30 11:34:43.629717 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.629695 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56f0158b-1fcb-4417-8182-995b72813052-config" pod "56f0158b-1fcb-4417-8182-995b72813052" (UID: "56f0158b-1fcb-4417-8182-995b72813052"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:34:43.641453 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.641400 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f0158b-1fcb-4417-8182-995b72813052-kube-api-access-s8cc2" pod "56f0158b-1fcb-4417-8182-995b72813052" (UID: "56f0158b-1fcb-4417-8182-995b72813052"). InnerVolumeSpecName "kube-api-access-s8cc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:34:43.644023 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.643994 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56f0158b-1fcb-4417-8182-995b72813052-tls" pod "56f0158b-1fcb-4417-8182-995b72813052" (UID: "56f0158b-1fcb-4417-8182-995b72813052"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:43.663029 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.663006 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" Oct 30 11:34:43.665793 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.665769 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl" Oct 30 11:34:43.675441 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.675408 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" Oct 30 11:34:43.718356 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.718322 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:43.730257 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.730216 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-kube-rbac-proxy-config\") pod \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\" (UID: \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\") " Oct 30 11:34:43.730412 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.730279 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-prometheus-ca\") pod \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\" (UID: \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\") " Oct 30 11:34:43.730412 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.730314 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34d3af29-0917-4d1d-934d-fbab9aa8f38b-kube-api-access-tzhtq\" (UniqueName: \"kubernetes.io/projected/34d3af29-0917-4d1d-934d-fbab9aa8f38b-kube-api-access-tzhtq\") pod \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\" (UID: \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\") " Oct 30 11:34:43.730412 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.730352 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-kube-api-access-755qz\" (UniqueName: \"kubernetes.io/projected/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-kube-api-access-755qz\") pod \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\" (UID: \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\") " Oct 30 11:34:43.730412 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.730405 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-prometheus-client-cert\") pod \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\" (UID: \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\") " Oct 30 11:34:43.730669 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.730456 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34d3af29-0917-4d1d-934d-fbab9aa8f38b-config\" (UniqueName: \"kubernetes.io/configmap/34d3af29-0917-4d1d-934d-fbab9aa8f38b-config\") pod \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\" (UID: \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\") " Oct 30 11:34:43.730669 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.730507 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34d3af29-0917-4d1d-934d-fbab9aa8f38b-client-ca\" (UniqueName: \"kubernetes.io/configmap/34d3af29-0917-4d1d-934d-fbab9aa8f38b-client-ca\") pod \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\" (UID: \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\") " Oct 30 11:34:43.730669 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.730550 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-tls\" (UniqueName: \"kubernetes.io/secret/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-tls\") pod \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\" (UID: \"e79c96e2-ffe5-41d4-823c-b86b416aa4f8\") " Oct 30 11:34:43.730669 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.730583 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34d3af29-0917-4d1d-934d-fbab9aa8f38b-tls\" (UniqueName: \"kubernetes.io/secret/34d3af29-0917-4d1d-934d-fbab9aa8f38b-tls\") pod \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\" (UID: \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\") " Oct 30 11:34:43.730669 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.730612 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b029b579-df40-408b-a55b-34a9ac9a1f12-kube-api-access-lwgs4\" (UniqueName: \"kubernetes.io/projected/b029b579-df40-408b-a55b-34a9ac9a1f12-kube-api-access-lwgs4\") pod \"b029b579-df40-408b-a55b-34a9ac9a1f12\" (UID: \"b029b579-df40-408b-a55b-34a9ac9a1f12\") " Oct 30 11:34:43.730669 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.730650 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b029b579-df40-408b-a55b-34a9ac9a1f12-tls\" (UniqueName: \"kubernetes.io/secret/b029b579-df40-408b-a55b-34a9ac9a1f12-tls\") pod \"b029b579-df40-408b-a55b-34a9ac9a1f12\" (UID: \"b029b579-df40-408b-a55b-34a9ac9a1f12\") " Oct 30 11:34:43.730956 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.730698 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-prometheus-ca" pod "e79c96e2-ffe5-41d4-823c-b86b416aa4f8" (UID: "e79c96e2-ffe5-41d4-823c-b86b416aa4f8"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:34:43.730956 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.730711 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b029b579-df40-408b-a55b-34a9ac9a1f12-config\" (UniqueName: \"kubernetes.io/configmap/b029b579-df40-408b-a55b-34a9ac9a1f12-config\") pod \"b029b579-df40-408b-a55b-34a9ac9a1f12\" (UID: \"b029b579-df40-408b-a55b-34a9ac9a1f12\") " Oct 30 11:34:43.730956 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.730782 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34d3af29-0917-4d1d-934d-fbab9aa8f38b-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/34d3af29-0917-4d1d-934d-fbab9aa8f38b-prometheus-tls-ca\") pod \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\" (UID: \"34d3af29-0917-4d1d-934d-fbab9aa8f38b\") " Oct 30 11:34:43.731971 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.731318 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34d3af29-0917-4d1d-934d-fbab9aa8f38b-config" pod "34d3af29-0917-4d1d-934d-fbab9aa8f38b" (UID: "34d3af29-0917-4d1d-934d-fbab9aa8f38b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:34:43.732256 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.732216 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34d3af29-0917-4d1d-934d-fbab9aa8f38b-client-ca" pod "34d3af29-0917-4d1d-934d-fbab9aa8f38b" (UID: "34d3af29-0917-4d1d-934d-fbab9aa8f38b"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:34:43.732256 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.732226 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b029b579-df40-408b-a55b-34a9ac9a1f12-config" pod "b029b579-df40-408b-a55b-34a9ac9a1f12" (UID: "b029b579-df40-408b-a55b-34a9ac9a1f12"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:34:43.732479 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.732331 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-prometheus-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:43.732479 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.732351 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56f0158b-1fcb-4417-8182-995b72813052-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:43.732479 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.732365 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8cc2\" (UniqueName: \"kubernetes.io/projected/56f0158b-1fcb-4417-8182-995b72813052-kube-api-access-s8cc2\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:43.732479 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.732379 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/56f0158b-1fcb-4417-8182-995b72813052-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:43.732715 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.732640 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34d3af29-0917-4d1d-934d-fbab9aa8f38b-prometheus-tls-ca" pod "34d3af29-0917-4d1d-934d-fbab9aa8f38b" (UID: "34d3af29-0917-4d1d-934d-fbab9aa8f38b"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:34:43.735351 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.735308 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-kube-api-access-755qz" pod "e79c96e2-ffe5-41d4-823c-b86b416aa4f8" (UID: "e79c96e2-ffe5-41d4-823c-b86b416aa4f8"). InnerVolumeSpecName "kube-api-access-755qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:34:43.738899 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.738878 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b029b579-df40-408b-a55b-34a9ac9a1f12-kube-api-access-lwgs4" pod "b029b579-df40-408b-a55b-34a9ac9a1f12" (UID: "b029b579-df40-408b-a55b-34a9ac9a1f12"). InnerVolumeSpecName "kube-api-access-lwgs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:34:43.739022 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.738905 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d3af29-0917-4d1d-934d-fbab9aa8f38b-kube-api-access-tzhtq" pod "34d3af29-0917-4d1d-934d-fbab9aa8f38b" (UID: "34d3af29-0917-4d1d-934d-fbab9aa8f38b"). InnerVolumeSpecName "kube-api-access-tzhtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:34:43.739198 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.739161 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-prometheus-client-cert" pod "e79c96e2-ffe5-41d4-823c-b86b416aa4f8" (UID: "e79c96e2-ffe5-41d4-823c-b86b416aa4f8"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:43.739290 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.739267 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34d3af29-0917-4d1d-934d-fbab9aa8f38b-tls" pod "34d3af29-0917-4d1d-934d-fbab9aa8f38b" (UID: "34d3af29-0917-4d1d-934d-fbab9aa8f38b"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:43.740115 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.740033 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-tls" pod "e79c96e2-ffe5-41d4-823c-b86b416aa4f8" (UID: "e79c96e2-ffe5-41d4-823c-b86b416aa4f8"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:43.741063 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.740754 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b029b579-df40-408b-a55b-34a9ac9a1f12-tls" pod "b029b579-df40-408b-a55b-34a9ac9a1f12" (UID: "b029b579-df40-408b-a55b-34a9ac9a1f12"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:43.741387 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.741320 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-kube-rbac-proxy-config" pod "e79c96e2-ffe5-41d4-823c-b86b416aa4f8" (UID: "e79c96e2-ffe5-41d4-823c-b86b416aa4f8"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:43.768670 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.768633 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:34:43.768990 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.768958 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="f31b5c28-b789-4b2a-a1f6-64d77dffa481" containerName="thanos-sidecar" containerID="cri-o://78fa504f2cea0ded06c3699d3a6cb64372eb90e1f27d7fd1f324ee9ca1db8ec1" gracePeriod=600 Oct 30 11:34:43.768990 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.768958 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="f31b5c28-b789-4b2a-a1f6-64d77dffa481" containerName="prometheus" containerID="cri-o://36ec3031da98de911ccd0c0b74981aeeb4673cc0af6c46cdc4367d874530ce8f" gracePeriod=600 Oct 30 11:34:43.769143 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.768967 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="f31b5c28-b789-4b2a-a1f6-64d77dffa481" containerName="config-reloader" containerID="cri-o://df76779925e290ac771ceba64e602d0dc83eb576effab05870852f41c741d61e" gracePeriod=600 Oct 30 11:34:43.781413 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.781380 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:34:43.781667 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.781645 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="c34bf4cd-d232-45d9-8e12-40ea4859b874" containerName="alertmanager" containerID="cri-o://198f4fd92c27383cfbeeb2e5931a7ccbe9d492e2ad563c6e7486be5eda12b260" gracePeriod=120 Oct 30 11:34:43.781751 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.781727 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="c34bf4cd-d232-45d9-8e12-40ea4859b874" containerName="config-reloader" containerID="cri-o://27c0870e6bd9d5d9833b9129e75087f67efaaf7346eb5eb11559b8e43023464e" gracePeriod=120 Oct 30 11:34:43.833483 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.833393 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b029b579-df40-408b-a55b-34a9ac9a1f12-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:43.833483 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.833420 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/34d3af29-0917-4d1d-934d-fbab9aa8f38b-prometheus-tls-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:43.833483 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.833444 4289 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-kube-rbac-proxy-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:43.833483 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.833453 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzhtq\" (UniqueName: \"kubernetes.io/projected/34d3af29-0917-4d1d-934d-fbab9aa8f38b-kube-api-access-tzhtq\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:43.833483 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.833462 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-755qz\" (UniqueName: \"kubernetes.io/projected/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-kube-api-access-755qz\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:43.833483 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.833470 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-prometheus-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:43.833483 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.833478 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34d3af29-0917-4d1d-934d-fbab9aa8f38b-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:43.833483 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.833486 4289 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/34d3af29-0917-4d1d-934d-fbab9aa8f38b-client-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:43.833834 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.833497 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e79c96e2-ffe5-41d4-823c-b86b416aa4f8-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:43.833834 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.833505 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/34d3af29-0917-4d1d-934d-fbab9aa8f38b-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:43.833834 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.833512 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwgs4\" (UniqueName: \"kubernetes.io/projected/b029b579-df40-408b-a55b-34a9ac9a1f12-kube-api-access-lwgs4\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:43.833834 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.833520 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b029b579-df40-408b-a55b-34a9ac9a1f12-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:43.899234 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:43.899207 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:44.027753 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.027720 4289 generic.go:356] "Generic (PLEG): container finished" podID="c34bf4cd-d232-45d9-8e12-40ea4859b874" containerID="27c0870e6bd9d5d9833b9129e75087f67efaaf7346eb5eb11559b8e43023464e" exitCode=0 Oct 30 11:34:44.027911 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.027793 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c34bf4cd-d232-45d9-8e12-40ea4859b874","Type":"ContainerDied","Data":"27c0870e6bd9d5d9833b9129e75087f67efaaf7346eb5eb11559b8e43023464e"} Oct 30 11:34:44.029027 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.029000 4289 generic.go:356] "Generic (PLEG): container finished" podID="b029b579-df40-408b-a55b-34a9ac9a1f12" containerID="5d148cce2f4c85beee066a557df141368abb906617533a4b36462af5205de3e9" exitCode=0 Oct 30 11:34:44.029128 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.029057 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl" Oct 30 11:34:44.029188 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.029053 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl" event={"ID":"b029b579-df40-408b-a55b-34a9ac9a1f12","Type":"ContainerDied","Data":"5d148cce2f4c85beee066a557df141368abb906617533a4b36462af5205de3e9"} Oct 30 11:34:44.029188 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.029168 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl" event={"ID":"b029b579-df40-408b-a55b-34a9ac9a1f12","Type":"ContainerDied","Data":"d0d232a6d83274f930e83f901a5700059cf10f34d5b7d24e16c820e386af8c50"} Oct 30 11:34:44.029263 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.029209 4289 scope.go:122] "RemoveContainer" containerID="5d148cce2f4c85beee066a557df141368abb906617533a4b36462af5205de3e9" Oct 30 11:34:44.031145 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.031123 4289 generic.go:356] "Generic (PLEG): container finished" podID="56f0158b-1fcb-4417-8182-995b72813052" containerID="3871660646bb1363bcb0074f8a3d0170acd949158caf0d1f266bd062aabd7d4b" exitCode=0 Oct 30 11:34:44.031145 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.031141 4289 generic.go:356] "Generic (PLEG): container finished" podID="56f0158b-1fcb-4417-8182-995b72813052" containerID="e5b7276a8e6bd3c0bbb16389cb5e2a7f00e637c0452ad27b2daca5c5b020d974" exitCode=0 Oct 30 11:34:44.031308 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.031150 4289 generic.go:356] "Generic (PLEG): container finished" podID="56f0158b-1fcb-4417-8182-995b72813052" containerID="63062fde5c2275f5d6ec29953bf56dbd89bd14a059f8532840cc63146c3e564e" exitCode=0 Oct 30 11:34:44.031308 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.031184 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" Oct 30 11:34:44.031308 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.031209 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" event={"ID":"56f0158b-1fcb-4417-8182-995b72813052","Type":"ContainerDied","Data":"3871660646bb1363bcb0074f8a3d0170acd949158caf0d1f266bd062aabd7d4b"} Oct 30 11:34:44.031308 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.031276 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" event={"ID":"56f0158b-1fcb-4417-8182-995b72813052","Type":"ContainerDied","Data":"e5b7276a8e6bd3c0bbb16389cb5e2a7f00e637c0452ad27b2daca5c5b020d974"} Oct 30 11:34:44.031578 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.031320 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" event={"ID":"56f0158b-1fcb-4417-8182-995b72813052","Type":"ContainerDied","Data":"63062fde5c2275f5d6ec29953bf56dbd89bd14a059f8532840cc63146c3e564e"} Oct 30 11:34:44.031578 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.031358 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf" event={"ID":"56f0158b-1fcb-4417-8182-995b72813052","Type":"ContainerDied","Data":"1a278c59dfa053513dea07e163ceb4920328733b4c5c89f374409cd44da2cdb4"} Oct 30 11:34:44.034218 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.034198 4289 generic.go:356] "Generic (PLEG): container finished" podID="f31b5c28-b789-4b2a-a1f6-64d77dffa481" containerID="78fa504f2cea0ded06c3699d3a6cb64372eb90e1f27d7fd1f324ee9ca1db8ec1" exitCode=0 Oct 30 11:34:44.034337 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.034222 4289 generic.go:356] "Generic (PLEG): container finished" podID="f31b5c28-b789-4b2a-a1f6-64d77dffa481" containerID="df76779925e290ac771ceba64e602d0dc83eb576effab05870852f41c741d61e" exitCode=0 Oct 30 11:34:44.034337 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.034298 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f31b5c28-b789-4b2a-a1f6-64d77dffa481","Type":"ContainerDied","Data":"78fa504f2cea0ded06c3699d3a6cb64372eb90e1f27d7fd1f324ee9ca1db8ec1"} Oct 30 11:34:44.034483 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.034353 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f31b5c28-b789-4b2a-a1f6-64d77dffa481","Type":"ContainerDied","Data":"df76779925e290ac771ceba64e602d0dc83eb576effab05870852f41c741d61e"} Oct 30 11:34:44.035748 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.035728 4289 generic.go:356] "Generic (PLEG): container finished" podID="e79c96e2-ffe5-41d4-823c-b86b416aa4f8" containerID="a80081a532bd69bbbe9d10ea90c89797f6c80f847eef57bf73d5701ddc495782" exitCode=0 Oct 30 11:34:44.035856 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.035774 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" Oct 30 11:34:44.035856 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.035816 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" event={"ID":"e79c96e2-ffe5-41d4-823c-b86b416aa4f8","Type":"ContainerDied","Data":"a80081a532bd69bbbe9d10ea90c89797f6c80f847eef57bf73d5701ddc495782"} Oct 30 11:34:44.035971 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.035871 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78" event={"ID":"e79c96e2-ffe5-41d4-823c-b86b416aa4f8","Type":"ContainerDied","Data":"6b785a7c878e7f274102c1aeec8a137614cfe2f81c9eb707e82411ad47618296"} Oct 30 11:34:44.037896 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.037874 4289 generic.go:356] "Generic (PLEG): container finished" podID="34d3af29-0917-4d1d-934d-fbab9aa8f38b" containerID="7110f6b5f75631639d64c0ae8b802e0387a9f8ef7ae72fd0119e5cc9d624cf93" exitCode=0 Oct 30 11:34:44.037896 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.037896 4289 generic.go:356] "Generic (PLEG): container finished" podID="34d3af29-0917-4d1d-934d-fbab9aa8f38b" containerID="e8bb2d2ae94770aec5b86d7f8cda3aa22f418a641930f29d747a93d20695bcb9" exitCode=0 Oct 30 11:34:44.038042 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.037904 4289 generic.go:356] "Generic (PLEG): container finished" podID="34d3af29-0917-4d1d-934d-fbab9aa8f38b" containerID="9a36b800cf97aafb3beef4312edd8655cde00df871b74521c11abe998d8d89a6" exitCode=0 Oct 30 11:34:44.038042 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.037933 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" Oct 30 11:34:44.038042 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.037948 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" event={"ID":"34d3af29-0917-4d1d-934d-fbab9aa8f38b","Type":"ContainerDied","Data":"7110f6b5f75631639d64c0ae8b802e0387a9f8ef7ae72fd0119e5cc9d624cf93"} Oct 30 11:34:44.038042 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.037999 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" event={"ID":"34d3af29-0917-4d1d-934d-fbab9aa8f38b","Type":"ContainerDied","Data":"e8bb2d2ae94770aec5b86d7f8cda3aa22f418a641930f29d747a93d20695bcb9"} Oct 30 11:34:44.038042 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.038038 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" event={"ID":"34d3af29-0917-4d1d-934d-fbab9aa8f38b","Type":"ContainerDied","Data":"9a36b800cf97aafb3beef4312edd8655cde00df871b74521c11abe998d8d89a6"} Oct 30 11:34:44.038308 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.038076 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f" event={"ID":"34d3af29-0917-4d1d-934d-fbab9aa8f38b","Type":"ContainerDied","Data":"5a51d27c8d5e70a673c1facafae9f6164f8c4b8d510ae28cb69b3bc43d980dab"} Oct 30 11:34:44.039383 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.039362 4289 scope.go:122] "RemoveContainer" containerID="5d148cce2f4c85beee066a557df141368abb906617533a4b36462af5205de3e9" Oct 30 11:34:44.039699 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:44.039669 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d148cce2f4c85beee066a557df141368abb906617533a4b36462af5205de3e9\": container with ID starting with 5d148cce2f4c85beee066a557df141368abb906617533a4b36462af5205de3e9 not found: ID does not exist" containerID="5d148cce2f4c85beee066a557df141368abb906617533a4b36462af5205de3e9" Oct 30 11:34:44.039800 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.039710 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d148cce2f4c85beee066a557df141368abb906617533a4b36462af5205de3e9"} err="failed to get container status \"5d148cce2f4c85beee066a557df141368abb906617533a4b36462af5205de3e9\": rpc error: code = NotFound desc = could not find container \"5d148cce2f4c85beee066a557df141368abb906617533a4b36462af5205de3e9\": container with ID starting with 5d148cce2f4c85beee066a557df141368abb906617533a4b36462af5205de3e9 not found: ID does not exist" Oct 30 11:34:44.039800 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.039734 4289 scope.go:122] "RemoveContainer" containerID="3871660646bb1363bcb0074f8a3d0170acd949158caf0d1f266bd062aabd7d4b" Oct 30 11:34:44.050097 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.050075 4289 scope.go:122] "RemoveContainer" containerID="e5b7276a8e6bd3c0bbb16389cb5e2a7f00e637c0452ad27b2daca5c5b020d974" Oct 30 11:34:44.057505 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.057486 4289 scope.go:122] "RemoveContainer" containerID="63062fde5c2275f5d6ec29953bf56dbd89bd14a059f8532840cc63146c3e564e" Oct 30 11:34:44.064875 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.064859 4289 scope.go:122] "RemoveContainer" containerID="3871660646bb1363bcb0074f8a3d0170acd949158caf0d1f266bd062aabd7d4b" Oct 30 11:34:44.065142 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:44.065126 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3871660646bb1363bcb0074f8a3d0170acd949158caf0d1f266bd062aabd7d4b\": container with ID starting with 3871660646bb1363bcb0074f8a3d0170acd949158caf0d1f266bd062aabd7d4b not found: ID does not exist" containerID="3871660646bb1363bcb0074f8a3d0170acd949158caf0d1f266bd062aabd7d4b" Oct 30 11:34:44.065196 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.065150 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3871660646bb1363bcb0074f8a3d0170acd949158caf0d1f266bd062aabd7d4b"} err="failed to get container status \"3871660646bb1363bcb0074f8a3d0170acd949158caf0d1f266bd062aabd7d4b\": rpc error: code = NotFound desc = could not find container \"3871660646bb1363bcb0074f8a3d0170acd949158caf0d1f266bd062aabd7d4b\": container with ID starting with 3871660646bb1363bcb0074f8a3d0170acd949158caf0d1f266bd062aabd7d4b not found: ID does not exist" Oct 30 11:34:44.065196 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.065166 4289 scope.go:122] "RemoveContainer" containerID="e5b7276a8e6bd3c0bbb16389cb5e2a7f00e637c0452ad27b2daca5c5b020d974" Oct 30 11:34:44.065406 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:44.065392 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5b7276a8e6bd3c0bbb16389cb5e2a7f00e637c0452ad27b2daca5c5b020d974\": container with ID starting with e5b7276a8e6bd3c0bbb16389cb5e2a7f00e637c0452ad27b2daca5c5b020d974 not found: ID does not exist" containerID="e5b7276a8e6bd3c0bbb16389cb5e2a7f00e637c0452ad27b2daca5c5b020d974" Oct 30 11:34:44.065457 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.065412 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b7276a8e6bd3c0bbb16389cb5e2a7f00e637c0452ad27b2daca5c5b020d974"} err="failed to get container status \"e5b7276a8e6bd3c0bbb16389cb5e2a7f00e637c0452ad27b2daca5c5b020d974\": rpc error: code = NotFound desc = could not find container \"e5b7276a8e6bd3c0bbb16389cb5e2a7f00e637c0452ad27b2daca5c5b020d974\": container with ID starting with e5b7276a8e6bd3c0bbb16389cb5e2a7f00e637c0452ad27b2daca5c5b020d974 not found: ID does not exist" Oct 30 11:34:44.065503 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.065465 4289 scope.go:122] "RemoveContainer" containerID="63062fde5c2275f5d6ec29953bf56dbd89bd14a059f8532840cc63146c3e564e" Oct 30 11:34:44.065728 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:44.065706 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63062fde5c2275f5d6ec29953bf56dbd89bd14a059f8532840cc63146c3e564e\": container with ID starting with 63062fde5c2275f5d6ec29953bf56dbd89bd14a059f8532840cc63146c3e564e not found: ID does not exist" containerID="63062fde5c2275f5d6ec29953bf56dbd89bd14a059f8532840cc63146c3e564e" Oct 30 11:34:44.065798 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.065736 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63062fde5c2275f5d6ec29953bf56dbd89bd14a059f8532840cc63146c3e564e"} err="failed to get container status \"63062fde5c2275f5d6ec29953bf56dbd89bd14a059f8532840cc63146c3e564e\": rpc error: code = NotFound desc = could not find container \"63062fde5c2275f5d6ec29953bf56dbd89bd14a059f8532840cc63146c3e564e\": container with ID starting with 63062fde5c2275f5d6ec29953bf56dbd89bd14a059f8532840cc63146c3e564e not found: ID does not exist" Oct 30 11:34:44.065798 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.065756 4289 scope.go:122] "RemoveContainer" containerID="3871660646bb1363bcb0074f8a3d0170acd949158caf0d1f266bd062aabd7d4b" Oct 30 11:34:44.066014 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.065994 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3871660646bb1363bcb0074f8a3d0170acd949158caf0d1f266bd062aabd7d4b"} err="failed to get container status \"3871660646bb1363bcb0074f8a3d0170acd949158caf0d1f266bd062aabd7d4b\": rpc error: code = NotFound desc = could not find container \"3871660646bb1363bcb0074f8a3d0170acd949158caf0d1f266bd062aabd7d4b\": container with ID starting with 3871660646bb1363bcb0074f8a3d0170acd949158caf0d1f266bd062aabd7d4b not found: ID does not exist" Oct 30 11:34:44.066070 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.066015 4289 scope.go:122] "RemoveContainer" containerID="e5b7276a8e6bd3c0bbb16389cb5e2a7f00e637c0452ad27b2daca5c5b020d974" Oct 30 11:34:44.066270 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.066248 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b7276a8e6bd3c0bbb16389cb5e2a7f00e637c0452ad27b2daca5c5b020d974"} err="failed to get container status \"e5b7276a8e6bd3c0bbb16389cb5e2a7f00e637c0452ad27b2daca5c5b020d974\": rpc error: code = NotFound desc = could not find container \"e5b7276a8e6bd3c0bbb16389cb5e2a7f00e637c0452ad27b2daca5c5b020d974\": container with ID starting with e5b7276a8e6bd3c0bbb16389cb5e2a7f00e637c0452ad27b2daca5c5b020d974 not found: ID does not exist" Oct 30 11:34:44.066323 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.066275 4289 scope.go:122] "RemoveContainer" containerID="63062fde5c2275f5d6ec29953bf56dbd89bd14a059f8532840cc63146c3e564e" Oct 30 11:34:44.066541 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.066509 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63062fde5c2275f5d6ec29953bf56dbd89bd14a059f8532840cc63146c3e564e"} err="failed to get container status \"63062fde5c2275f5d6ec29953bf56dbd89bd14a059f8532840cc63146c3e564e\": rpc error: code = NotFound desc = could not find container \"63062fde5c2275f5d6ec29953bf56dbd89bd14a059f8532840cc63146c3e564e\": container with ID starting with 63062fde5c2275f5d6ec29953bf56dbd89bd14a059f8532840cc63146c3e564e not found: ID does not exist" Oct 30 11:34:44.066541 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.066533 4289 scope.go:122] "RemoveContainer" containerID="3871660646bb1363bcb0074f8a3d0170acd949158caf0d1f266bd062aabd7d4b" Oct 30 11:34:44.066790 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.066754 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3871660646bb1363bcb0074f8a3d0170acd949158caf0d1f266bd062aabd7d4b"} err="failed to get container status \"3871660646bb1363bcb0074f8a3d0170acd949158caf0d1f266bd062aabd7d4b\": rpc error: code = NotFound desc = could not find container \"3871660646bb1363bcb0074f8a3d0170acd949158caf0d1f266bd062aabd7d4b\": container with ID starting with 3871660646bb1363bcb0074f8a3d0170acd949158caf0d1f266bd062aabd7d4b not found: ID does not exist" Oct 30 11:34:44.066790 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.066777 4289 scope.go:122] "RemoveContainer" containerID="e5b7276a8e6bd3c0bbb16389cb5e2a7f00e637c0452ad27b2daca5c5b020d974" Oct 30 11:34:44.066953 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.066810 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:44.067049 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.067029 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b7276a8e6bd3c0bbb16389cb5e2a7f00e637c0452ad27b2daca5c5b020d974"} err="failed to get container status \"e5b7276a8e6bd3c0bbb16389cb5e2a7f00e637c0452ad27b2daca5c5b020d974\": rpc error: code = NotFound desc = could not find container \"e5b7276a8e6bd3c0bbb16389cb5e2a7f00e637c0452ad27b2daca5c5b020d974\": container with ID starting with e5b7276a8e6bd3c0bbb16389cb5e2a7f00e637c0452ad27b2daca5c5b020d974 not found: ID does not exist" Oct 30 11:34:44.067093 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.067049 4289 scope.go:122] "RemoveContainer" containerID="63062fde5c2275f5d6ec29953bf56dbd89bd14a059f8532840cc63146c3e564e" Oct 30 11:34:44.067280 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.067262 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63062fde5c2275f5d6ec29953bf56dbd89bd14a059f8532840cc63146c3e564e"} err="failed to get container status \"63062fde5c2275f5d6ec29953bf56dbd89bd14a059f8532840cc63146c3e564e\": rpc error: code = NotFound desc = could not find container \"63062fde5c2275f5d6ec29953bf56dbd89bd14a059f8532840cc63146c3e564e\": container with ID starting with 63062fde5c2275f5d6ec29953bf56dbd89bd14a059f8532840cc63146c3e564e not found: ID does not exist" Oct 30 11:34:44.067325 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.067280 4289 scope.go:122] "RemoveContainer" containerID="a80081a532bd69bbbe9d10ea90c89797f6c80f847eef57bf73d5701ddc495782" Oct 30 11:34:44.074502 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.074479 4289 scope.go:122] "RemoveContainer" containerID="a80081a532bd69bbbe9d10ea90c89797f6c80f847eef57bf73d5701ddc495782" Oct 30 11:34:44.074786 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:44.074763 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80081a532bd69bbbe9d10ea90c89797f6c80f847eef57bf73d5701ddc495782\": container with ID starting with a80081a532bd69bbbe9d10ea90c89797f6c80f847eef57bf73d5701ddc495782 not found: ID does not exist" containerID="a80081a532bd69bbbe9d10ea90c89797f6c80f847eef57bf73d5701ddc495782" Oct 30 11:34:44.074858 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.074790 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80081a532bd69bbbe9d10ea90c89797f6c80f847eef57bf73d5701ddc495782"} err="failed to get container status \"a80081a532bd69bbbe9d10ea90c89797f6c80f847eef57bf73d5701ddc495782\": rpc error: code = NotFound desc = could not find container \"a80081a532bd69bbbe9d10ea90c89797f6c80f847eef57bf73d5701ddc495782\": container with ID starting with a80081a532bd69bbbe9d10ea90c89797f6c80f847eef57bf73d5701ddc495782 not found: ID does not exist" Oct 30 11:34:44.074858 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.074804 4289 scope.go:122] "RemoveContainer" containerID="7110f6b5f75631639d64c0ae8b802e0387a9f8ef7ae72fd0119e5cc9d624cf93" Oct 30 11:34:44.082201 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.082185 4289 scope.go:122] "RemoveContainer" containerID="e8bb2d2ae94770aec5b86d7f8cda3aa22f418a641930f29d747a93d20695bcb9" Oct 30 11:34:44.089077 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.089063 4289 scope.go:122] "RemoveContainer" containerID="9a36b800cf97aafb3beef4312edd8655cde00df871b74521c11abe998d8d89a6" Oct 30 11:34:44.096067 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.096051 4289 scope.go:122] "RemoveContainer" containerID="7110f6b5f75631639d64c0ae8b802e0387a9f8ef7ae72fd0119e5cc9d624cf93" Oct 30 11:34:44.096344 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:44.096325 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7110f6b5f75631639d64c0ae8b802e0387a9f8ef7ae72fd0119e5cc9d624cf93\": container with ID starting with 7110f6b5f75631639d64c0ae8b802e0387a9f8ef7ae72fd0119e5cc9d624cf93 not found: ID does not exist" containerID="7110f6b5f75631639d64c0ae8b802e0387a9f8ef7ae72fd0119e5cc9d624cf93" Oct 30 11:34:44.096421 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.096354 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7110f6b5f75631639d64c0ae8b802e0387a9f8ef7ae72fd0119e5cc9d624cf93"} err="failed to get container status \"7110f6b5f75631639d64c0ae8b802e0387a9f8ef7ae72fd0119e5cc9d624cf93\": rpc error: code = NotFound desc = could not find container \"7110f6b5f75631639d64c0ae8b802e0387a9f8ef7ae72fd0119e5cc9d624cf93\": container with ID starting with 7110f6b5f75631639d64c0ae8b802e0387a9f8ef7ae72fd0119e5cc9d624cf93 not found: ID does not exist" Oct 30 11:34:44.096421 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.096378 4289 scope.go:122] "RemoveContainer" containerID="e8bb2d2ae94770aec5b86d7f8cda3aa22f418a641930f29d747a93d20695bcb9" Oct 30 11:34:44.096675 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:44.096657 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8bb2d2ae94770aec5b86d7f8cda3aa22f418a641930f29d747a93d20695bcb9\": container with ID starting with e8bb2d2ae94770aec5b86d7f8cda3aa22f418a641930f29d747a93d20695bcb9 not found: ID does not exist" containerID="e8bb2d2ae94770aec5b86d7f8cda3aa22f418a641930f29d747a93d20695bcb9" Oct 30 11:34:44.096720 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.096684 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8bb2d2ae94770aec5b86d7f8cda3aa22f418a641930f29d747a93d20695bcb9"} err="failed to get container status \"e8bb2d2ae94770aec5b86d7f8cda3aa22f418a641930f29d747a93d20695bcb9\": rpc error: code = NotFound desc = could not find container \"e8bb2d2ae94770aec5b86d7f8cda3aa22f418a641930f29d747a93d20695bcb9\": container with ID starting with e8bb2d2ae94770aec5b86d7f8cda3aa22f418a641930f29d747a93d20695bcb9 not found: ID does not exist" Oct 30 11:34:44.096720 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.096700 4289 scope.go:122] "RemoveContainer" containerID="9a36b800cf97aafb3beef4312edd8655cde00df871b74521c11abe998d8d89a6" Oct 30 11:34:44.096915 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:44.096897 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a36b800cf97aafb3beef4312edd8655cde00df871b74521c11abe998d8d89a6\": container with ID starting with 9a36b800cf97aafb3beef4312edd8655cde00df871b74521c11abe998d8d89a6 not found: ID does not exist" containerID="9a36b800cf97aafb3beef4312edd8655cde00df871b74521c11abe998d8d89a6" Oct 30 11:34:44.096957 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.096919 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a36b800cf97aafb3beef4312edd8655cde00df871b74521c11abe998d8d89a6"} err="failed to get container status \"9a36b800cf97aafb3beef4312edd8655cde00df871b74521c11abe998d8d89a6\": rpc error: code = NotFound desc = could not find container \"9a36b800cf97aafb3beef4312edd8655cde00df871b74521c11abe998d8d89a6\": container with ID starting with 9a36b800cf97aafb3beef4312edd8655cde00df871b74521c11abe998d8d89a6 not found: ID does not exist" Oct 30 11:34:44.096957 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.096931 4289 scope.go:122] "RemoveContainer" containerID="7110f6b5f75631639d64c0ae8b802e0387a9f8ef7ae72fd0119e5cc9d624cf93" Oct 30 11:34:44.097124 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.097107 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7110f6b5f75631639d64c0ae8b802e0387a9f8ef7ae72fd0119e5cc9d624cf93"} err="failed to get container status \"7110f6b5f75631639d64c0ae8b802e0387a9f8ef7ae72fd0119e5cc9d624cf93\": rpc error: code = NotFound desc = could not find container \"7110f6b5f75631639d64c0ae8b802e0387a9f8ef7ae72fd0119e5cc9d624cf93\": container with ID starting with 7110f6b5f75631639d64c0ae8b802e0387a9f8ef7ae72fd0119e5cc9d624cf93 not found: ID does not exist" Oct 30 11:34:44.097166 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.097124 4289 scope.go:122] "RemoveContainer" containerID="e8bb2d2ae94770aec5b86d7f8cda3aa22f418a641930f29d747a93d20695bcb9" Oct 30 11:34:44.097326 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.097310 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8bb2d2ae94770aec5b86d7f8cda3aa22f418a641930f29d747a93d20695bcb9"} err="failed to get container status \"e8bb2d2ae94770aec5b86d7f8cda3aa22f418a641930f29d747a93d20695bcb9\": rpc error: code = NotFound desc = could not find container \"e8bb2d2ae94770aec5b86d7f8cda3aa22f418a641930f29d747a93d20695bcb9\": container with ID starting with e8bb2d2ae94770aec5b86d7f8cda3aa22f418a641930f29d747a93d20695bcb9 not found: ID does not exist" Oct 30 11:34:44.097326 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.097326 4289 scope.go:122] "RemoveContainer" containerID="9a36b800cf97aafb3beef4312edd8655cde00df871b74521c11abe998d8d89a6" Oct 30 11:34:44.098453 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.097877 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a36b800cf97aafb3beef4312edd8655cde00df871b74521c11abe998d8d89a6"} err="failed to get container status \"9a36b800cf97aafb3beef4312edd8655cde00df871b74521c11abe998d8d89a6\": rpc error: code = NotFound desc = could not find container \"9a36b800cf97aafb3beef4312edd8655cde00df871b74521c11abe998d8d89a6\": container with ID starting with 9a36b800cf97aafb3beef4312edd8655cde00df871b74521c11abe998d8d89a6 not found: ID does not exist" Oct 30 11:34:44.098453 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.097920 4289 scope.go:122] "RemoveContainer" containerID="7110f6b5f75631639d64c0ae8b802e0387a9f8ef7ae72fd0119e5cc9d624cf93" Oct 30 11:34:44.104649 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.104621 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7110f6b5f75631639d64c0ae8b802e0387a9f8ef7ae72fd0119e5cc9d624cf93"} err="failed to get container status \"7110f6b5f75631639d64c0ae8b802e0387a9f8ef7ae72fd0119e5cc9d624cf93\": rpc error: code = NotFound desc = could not find container \"7110f6b5f75631639d64c0ae8b802e0387a9f8ef7ae72fd0119e5cc9d624cf93\": container with ID starting with 7110f6b5f75631639d64c0ae8b802e0387a9f8ef7ae72fd0119e5cc9d624cf93 not found: ID does not exist" Oct 30 11:34:44.104649 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.104648 4289 scope.go:122] "RemoveContainer" containerID="e8bb2d2ae94770aec5b86d7f8cda3aa22f418a641930f29d747a93d20695bcb9" Oct 30 11:34:44.104928 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.104904 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8bb2d2ae94770aec5b86d7f8cda3aa22f418a641930f29d747a93d20695bcb9"} err="failed to get container status \"e8bb2d2ae94770aec5b86d7f8cda3aa22f418a641930f29d747a93d20695bcb9\": rpc error: code = NotFound desc = could not find container \"e8bb2d2ae94770aec5b86d7f8cda3aa22f418a641930f29d747a93d20695bcb9\": container with ID starting with e8bb2d2ae94770aec5b86d7f8cda3aa22f418a641930f29d747a93d20695bcb9 not found: ID does not exist" Oct 30 11:34:44.104995 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.104928 4289 scope.go:122] "RemoveContainer" containerID="9a36b800cf97aafb3beef4312edd8655cde00df871b74521c11abe998d8d89a6" Oct 30 11:34:44.105155 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.105135 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a36b800cf97aafb3beef4312edd8655cde00df871b74521c11abe998d8d89a6"} err="failed to get container status \"9a36b800cf97aafb3beef4312edd8655cde00df871b74521c11abe998d8d89a6\": rpc error: code = NotFound desc = could not find container \"9a36b800cf97aafb3beef4312edd8655cde00df871b74521c11abe998d8d89a6\": container with ID starting with 9a36b800cf97aafb3beef4312edd8655cde00df871b74521c11abe998d8d89a6 not found: ID does not exist" Oct 30 11:34:44.111641 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.111623 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf"] Oct 30 11:34:44.115191 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.115171 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-d6cwf"] Oct 30 11:34:44.152933 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.152901 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl"] Oct 30 11:34:44.157085 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.157062 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-wfmrl"] Oct 30 11:34:44.191698 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.191676 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78"] Oct 30 11:34:44.195063 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.195040 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-hdp78"] Oct 30 11:34:44.230803 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.230776 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f"] Oct 30 11:34:44.234575 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.234554 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-9w24f"] Oct 30 11:34:44.389873 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.389845 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:34:44.390259 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.390116 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="1808eff5-6fef-47ae-b389-2f8e0c4859c4" containerName="otc-container" containerID="cri-o://9f0fee691b66ea94933da307d7d13137fe95e3ba454f734a443d765491fb3675" gracePeriod=30 Oct 30 11:34:44.468581 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.468546 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:44.637926 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.637895 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:44.646507 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.646445 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:44.661463 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.661439 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:44.743387 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.743345 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c34bf4cd-d232-45d9-8e12-40ea4859b874-config-volume\" (UniqueName: \"kubernetes.io/secret/c34bf4cd-d232-45d9-8e12-40ea4859b874-config-volume\") pod \"c34bf4cd-d232-45d9-8e12-40ea4859b874\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " Oct 30 11:34:44.743581 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.743442 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34bf4cd-d232-45d9-8e12-40ea4859b874-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c34bf4cd-d232-45d9-8e12-40ea4859b874-alertmanager-data-science-monitoringstack-db\") pod \"c34bf4cd-d232-45d9-8e12-40ea4859b874\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " Oct 30 11:34:44.743581 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.743504 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c34bf4cd-d232-45d9-8e12-40ea4859b874-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c34bf4cd-d232-45d9-8e12-40ea4859b874-cluster-tls-config\") pod \"c34bf4cd-d232-45d9-8e12-40ea4859b874\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " Oct 30 11:34:44.743581 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.743554 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1808eff5-6fef-47ae-b389-2f8e0c4859c4-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1808eff5-6fef-47ae-b389-2f8e0c4859c4-monitoring-tls-certs\") pod \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\" (UID: \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\") " Oct 30 11:34:44.743750 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.743587 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34bf4cd-d232-45d9-8e12-40ea4859b874-kube-api-access-hgjvb\" (UniqueName: \"kubernetes.io/projected/c34bf4cd-d232-45d9-8e12-40ea4859b874-kube-api-access-hgjvb\") pod \"c34bf4cd-d232-45d9-8e12-40ea4859b874\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " Oct 30 11:34:44.743750 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.743614 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1808eff5-6fef-47ae-b389-2f8e0c4859c4-tls-certs\" (UniqueName: \"kubernetes.io/secret/1808eff5-6fef-47ae-b389-2f8e0c4859c4-tls-certs\") pod \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\" (UID: \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\") " Oct 30 11:34:44.743750 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.743642 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1808eff5-6fef-47ae-b389-2f8e0c4859c4-kube-api-access-m4sx5\" (UniqueName: \"kubernetes.io/projected/1808eff5-6fef-47ae-b389-2f8e0c4859c4-kube-api-access-m4sx5\") pod \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\" (UID: \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\") " Oct 30 11:34:44.743750 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.743689 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1808eff5-6fef-47ae-b389-2f8e0c4859c4-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1808eff5-6fef-47ae-b389-2f8e0c4859c4-data-science-collector-ta-client-cert\") pod \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\" (UID: \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\") " Oct 30 11:34:44.743750 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.743719 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34bf4cd-d232-45d9-8e12-40ea4859b874-tls-assets\" (UniqueName: \"kubernetes.io/projected/c34bf4cd-d232-45d9-8e12-40ea4859b874-tls-assets\") pod \"c34bf4cd-d232-45d9-8e12-40ea4859b874\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " Oct 30 11:34:44.744026 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.743751 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1808eff5-6fef-47ae-b389-2f8e0c4859c4-otc-internal\" (UniqueName: \"kubernetes.io/configmap/1808eff5-6fef-47ae-b389-2f8e0c4859c4-otc-internal\") pod \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\" (UID: \"1808eff5-6fef-47ae-b389-2f8e0c4859c4\") " Oct 30 11:34:44.744026 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.743787 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c34bf4cd-d232-45d9-8e12-40ea4859b874-web-config\" (UniqueName: \"kubernetes.io/secret/c34bf4cd-d232-45d9-8e12-40ea4859b874-web-config\") pod \"c34bf4cd-d232-45d9-8e12-40ea4859b874\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " Oct 30 11:34:44.744026 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.743843 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34bf4cd-d232-45d9-8e12-40ea4859b874-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c34bf4cd-d232-45d9-8e12-40ea4859b874-config-out\") pod \"c34bf4cd-d232-45d9-8e12-40ea4859b874\" (UID: \"c34bf4cd-d232-45d9-8e12-40ea4859b874\") " Oct 30 11:34:44.744026 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.743893 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34bf4cd-d232-45d9-8e12-40ea4859b874-alertmanager-data-science-monitoringstack-db" pod "c34bf4cd-d232-45d9-8e12-40ea4859b874" (UID: "c34bf4cd-d232-45d9-8e12-40ea4859b874"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:34:44.744241 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.744194 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1808eff5-6fef-47ae-b389-2f8e0c4859c4-otc-internal" pod "1808eff5-6fef-47ae-b389-2f8e0c4859c4" (UID: "1808eff5-6fef-47ae-b389-2f8e0c4859c4"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:34:44.744328 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.744295 4289 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c34bf4cd-d232-45d9-8e12-40ea4859b874-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:44.747506 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.747478 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34bf4cd-d232-45d9-8e12-40ea4859b874-config-out" pod "c34bf4cd-d232-45d9-8e12-40ea4859b874" (UID: "c34bf4cd-d232-45d9-8e12-40ea4859b874"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:34:44.749066 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.749040 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c34bf4cd-d232-45d9-8e12-40ea4859b874-config-volume" pod "c34bf4cd-d232-45d9-8e12-40ea4859b874" (UID: "c34bf4cd-d232-45d9-8e12-40ea4859b874"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:44.749066 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.749052 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1808eff5-6fef-47ae-b389-2f8e0c4859c4-kube-api-access-m4sx5" pod "1808eff5-6fef-47ae-b389-2f8e0c4859c4" (UID: "1808eff5-6fef-47ae-b389-2f8e0c4859c4"). InnerVolumeSpecName "kube-api-access-m4sx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:34:44.749246 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.749078 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34bf4cd-d232-45d9-8e12-40ea4859b874-kube-api-access-hgjvb" pod "c34bf4cd-d232-45d9-8e12-40ea4859b874" (UID: "c34bf4cd-d232-45d9-8e12-40ea4859b874"). InnerVolumeSpecName "kube-api-access-hgjvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:34:44.749246 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.749094 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1808eff5-6fef-47ae-b389-2f8e0c4859c4-monitoring-tls-certs" pod "1808eff5-6fef-47ae-b389-2f8e0c4859c4" (UID: "1808eff5-6fef-47ae-b389-2f8e0c4859c4"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:44.749739 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.749713 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34bf4cd-d232-45d9-8e12-40ea4859b874-tls-assets" pod "c34bf4cd-d232-45d9-8e12-40ea4859b874" (UID: "c34bf4cd-d232-45d9-8e12-40ea4859b874"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:34:44.749856 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.749779 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1808eff5-6fef-47ae-b389-2f8e0c4859c4-tls-certs" pod "1808eff5-6fef-47ae-b389-2f8e0c4859c4" (UID: "1808eff5-6fef-47ae-b389-2f8e0c4859c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:44.750570 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.750532 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1808eff5-6fef-47ae-b389-2f8e0c4859c4-data-science-collector-ta-client-cert" pod "1808eff5-6fef-47ae-b389-2f8e0c4859c4" (UID: "1808eff5-6fef-47ae-b389-2f8e0c4859c4"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:44.752137 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.752118 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c34bf4cd-d232-45d9-8e12-40ea4859b874-cluster-tls-config" pod "c34bf4cd-d232-45d9-8e12-40ea4859b874" (UID: "c34bf4cd-d232-45d9-8e12-40ea4859b874"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:44.762220 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.762192 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c34bf4cd-d232-45d9-8e12-40ea4859b874-web-config" pod "c34bf4cd-d232-45d9-8e12-40ea4859b874" (UID: "c34bf4cd-d232-45d9-8e12-40ea4859b874"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:44.845108 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.845073 4289 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c34bf4cd-d232-45d9-8e12-40ea4859b874-cluster-tls-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:44.845108 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.845104 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1808eff5-6fef-47ae-b389-2f8e0c4859c4-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:44.845108 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.845114 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgjvb\" (UniqueName: \"kubernetes.io/projected/c34bf4cd-d232-45d9-8e12-40ea4859b874-kube-api-access-hgjvb\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:44.845323 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.845124 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1808eff5-6fef-47ae-b389-2f8e0c4859c4-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:44.845323 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.845135 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4sx5\" (UniqueName: \"kubernetes.io/projected/1808eff5-6fef-47ae-b389-2f8e0c4859c4-kube-api-access-m4sx5\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:44.845323 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.845144 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1808eff5-6fef-47ae-b389-2f8e0c4859c4-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:44.845323 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.845152 4289 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c34bf4cd-d232-45d9-8e12-40ea4859b874-tls-assets\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:44.845323 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.845160 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1808eff5-6fef-47ae-b389-2f8e0c4859c4-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:44.845323 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.845167 4289 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c34bf4cd-d232-45d9-8e12-40ea4859b874-web-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:44.845323 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.845174 4289 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c34bf4cd-d232-45d9-8e12-40ea4859b874-config-out\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:44.845323 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:44.845181 4289 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c34bf4cd-d232-45d9-8e12-40ea4859b874-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:45.049011 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.048974 4289 generic.go:356] "Generic (PLEG): container finished" podID="c34bf4cd-d232-45d9-8e12-40ea4859b874" containerID="198f4fd92c27383cfbeeb2e5931a7ccbe9d492e2ad563c6e7486be5eda12b260" exitCode=0 Oct 30 11:34:45.049206 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.049050 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:34:45.049206 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.049051 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c34bf4cd-d232-45d9-8e12-40ea4859b874","Type":"ContainerDied","Data":"198f4fd92c27383cfbeeb2e5931a7ccbe9d492e2ad563c6e7486be5eda12b260"} Oct 30 11:34:45.049206 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.049191 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c34bf4cd-d232-45d9-8e12-40ea4859b874","Type":"ContainerDied","Data":"0bf331c51f4a1ed2514ea18ef6482c39e75ec8a667df17fb7923b3caf5aeb183"} Oct 30 11:34:45.049360 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.049236 4289 scope.go:122] "RemoveContainer" containerID="27c0870e6bd9d5d9833b9129e75087f67efaaf7346eb5eb11559b8e43023464e" Oct 30 11:34:45.050488 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.050468 4289 generic.go:356] "Generic (PLEG): container finished" podID="1808eff5-6fef-47ae-b389-2f8e0c4859c4" containerID="9f0fee691b66ea94933da307d7d13137fe95e3ba454f734a443d765491fb3675" exitCode=0 Oct 30 11:34:45.050595 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.050507 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1808eff5-6fef-47ae-b389-2f8e0c4859c4","Type":"ContainerDied","Data":"9f0fee691b66ea94933da307d7d13137fe95e3ba454f734a443d765491fb3675"} Oct 30 11:34:45.050595 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.050542 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1808eff5-6fef-47ae-b389-2f8e0c4859c4","Type":"ContainerDied","Data":"84f07e023a3077521a3635ed8817bcca6df1f52e0158658d39d57102e99e7621"} Oct 30 11:34:45.050595 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.050574 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:34:45.058885 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.058869 4289 scope.go:122] "RemoveContainer" containerID="198f4fd92c27383cfbeeb2e5931a7ccbe9d492e2ad563c6e7486be5eda12b260" Oct 30 11:34:45.066885 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.066868 4289 scope.go:122] "RemoveContainer" containerID="d22a0e114d493b8afe9fc08f4ec40c048a1c1bffcfadb06f660baa005269d04d" Oct 30 11:34:45.074514 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.074496 4289 scope.go:122] "RemoveContainer" containerID="27c0870e6bd9d5d9833b9129e75087f67efaaf7346eb5eb11559b8e43023464e" Oct 30 11:34:45.074782 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:45.074762 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c0870e6bd9d5d9833b9129e75087f67efaaf7346eb5eb11559b8e43023464e\": container with ID starting with 27c0870e6bd9d5d9833b9129e75087f67efaaf7346eb5eb11559b8e43023464e not found: ID does not exist" containerID="27c0870e6bd9d5d9833b9129e75087f67efaaf7346eb5eb11559b8e43023464e" Oct 30 11:34:45.074839 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.074795 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c0870e6bd9d5d9833b9129e75087f67efaaf7346eb5eb11559b8e43023464e"} err="failed to get container status \"27c0870e6bd9d5d9833b9129e75087f67efaaf7346eb5eb11559b8e43023464e\": rpc error: code = NotFound desc = could not find container \"27c0870e6bd9d5d9833b9129e75087f67efaaf7346eb5eb11559b8e43023464e\": container with ID starting with 27c0870e6bd9d5d9833b9129e75087f67efaaf7346eb5eb11559b8e43023464e not found: ID does not exist" Oct 30 11:34:45.074839 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.074814 4289 scope.go:122] "RemoveContainer" containerID="198f4fd92c27383cfbeeb2e5931a7ccbe9d492e2ad563c6e7486be5eda12b260" Oct 30 11:34:45.075088 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:45.075070 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"198f4fd92c27383cfbeeb2e5931a7ccbe9d492e2ad563c6e7486be5eda12b260\": container with ID starting with 198f4fd92c27383cfbeeb2e5931a7ccbe9d492e2ad563c6e7486be5eda12b260 not found: ID does not exist" containerID="198f4fd92c27383cfbeeb2e5931a7ccbe9d492e2ad563c6e7486be5eda12b260" Oct 30 11:34:45.075138 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.075095 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198f4fd92c27383cfbeeb2e5931a7ccbe9d492e2ad563c6e7486be5eda12b260"} err="failed to get container status \"198f4fd92c27383cfbeeb2e5931a7ccbe9d492e2ad563c6e7486be5eda12b260\": rpc error: code = NotFound desc = could not find container \"198f4fd92c27383cfbeeb2e5931a7ccbe9d492e2ad563c6e7486be5eda12b260\": container with ID starting with 198f4fd92c27383cfbeeb2e5931a7ccbe9d492e2ad563c6e7486be5eda12b260 not found: ID does not exist" Oct 30 11:34:45.075138 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.075110 4289 scope.go:122] "RemoveContainer" containerID="d22a0e114d493b8afe9fc08f4ec40c048a1c1bffcfadb06f660baa005269d04d" Oct 30 11:34:45.075310 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:45.075294 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d22a0e114d493b8afe9fc08f4ec40c048a1c1bffcfadb06f660baa005269d04d\": container with ID starting with d22a0e114d493b8afe9fc08f4ec40c048a1c1bffcfadb06f660baa005269d04d not found: ID does not exist" containerID="d22a0e114d493b8afe9fc08f4ec40c048a1c1bffcfadb06f660baa005269d04d" Oct 30 11:34:45.075352 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.075315 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d22a0e114d493b8afe9fc08f4ec40c048a1c1bffcfadb06f660baa005269d04d"} err="failed to get container status \"d22a0e114d493b8afe9fc08f4ec40c048a1c1bffcfadb06f660baa005269d04d\": rpc error: code = NotFound desc = could not find container \"d22a0e114d493b8afe9fc08f4ec40c048a1c1bffcfadb06f660baa005269d04d\": container with ID starting with d22a0e114d493b8afe9fc08f4ec40c048a1c1bffcfadb06f660baa005269d04d not found: ID does not exist" Oct 30 11:34:45.075352 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.075327 4289 scope.go:122] "RemoveContainer" containerID="9f0fee691b66ea94933da307d7d13137fe95e3ba454f734a443d765491fb3675" Oct 30 11:34:45.082319 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.082302 4289 scope.go:122] "RemoveContainer" containerID="9f0fee691b66ea94933da307d7d13137fe95e3ba454f734a443d765491fb3675" Oct 30 11:34:45.082560 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:45.082542 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f0fee691b66ea94933da307d7d13137fe95e3ba454f734a443d765491fb3675\": container with ID starting with 9f0fee691b66ea94933da307d7d13137fe95e3ba454f734a443d765491fb3675 not found: ID does not exist" containerID="9f0fee691b66ea94933da307d7d13137fe95e3ba454f734a443d765491fb3675" Oct 30 11:34:45.082621 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.082566 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0fee691b66ea94933da307d7d13137fe95e3ba454f734a443d765491fb3675"} err="failed to get container status \"9f0fee691b66ea94933da307d7d13137fe95e3ba454f734a443d765491fb3675\": rpc error: code = NotFound desc = could not find container \"9f0fee691b66ea94933da307d7d13137fe95e3ba454f734a443d765491fb3675\": container with ID starting with 9f0fee691b66ea94933da307d7d13137fe95e3ba454f734a443d765491fb3675 not found: ID does not exist" Oct 30 11:34:45.102261 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.102236 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:34:45.106252 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.106231 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:34:45.140967 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.140938 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:34:45.145673 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.145649 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:34:45.351573 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.351492 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:45.520678 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.520646 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:34:45.625822 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.625751 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1808eff5-6fef-47ae-b389-2f8e0c4859c4" path="/var/lib/kubelet/pods/1808eff5-6fef-47ae-b389-2f8e0c4859c4/volumes" Oct 30 11:34:45.626135 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.626122 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34d3af29-0917-4d1d-934d-fbab9aa8f38b" path="/var/lib/kubelet/pods/34d3af29-0917-4d1d-934d-fbab9aa8f38b/volumes" Oct 30 11:34:45.626553 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.626541 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56f0158b-1fcb-4417-8182-995b72813052" path="/var/lib/kubelet/pods/56f0158b-1fcb-4417-8182-995b72813052/volumes" Oct 30 11:34:45.627270 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.627256 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b029b579-df40-408b-a55b-34a9ac9a1f12" path="/var/lib/kubelet/pods/b029b579-df40-408b-a55b-34a9ac9a1f12/volumes" Oct 30 11:34:45.627844 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.627817 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c34bf4cd-d232-45d9-8e12-40ea4859b874" path="/var/lib/kubelet/pods/c34bf4cd-d232-45d9-8e12-40ea4859b874/volumes" Oct 30 11:34:45.628543 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:45.628526 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e79c96e2-ffe5-41d4-823c-b86b416aa4f8" path="/var/lib/kubelet/pods/e79c96e2-ffe5-41d4-823c-b86b416aa4f8/volumes" Oct 30 11:34:46.972904 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:46.972861 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="f31b5c28-b789-4b2a-a1f6-64d77dffa481" containerName="prometheus" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 11:34:51.973182 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:51.973146 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="f31b5c28-b789-4b2a-a1f6-64d77dffa481" containerName="prometheus" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 11:34:55.617755 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.617733 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:55.737786 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.737761 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " Oct 30 11:34:55.737944 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.737809 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-secret-prometheus-web-tls-ca\") pod \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " Oct 30 11:34:55.737944 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.737832 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f31b5c28-b789-4b2a-a1f6-64d77dffa481-tls-assets\" (UniqueName: \"kubernetes.io/projected/f31b5c28-b789-4b2a-a1f6-64d77dffa481-tls-assets\") pod \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " Oct 30 11:34:55.737944 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.737865 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f31b5c28-b789-4b2a-a1f6-64d77dffa481-config-out\" (UniqueName: \"kubernetes.io/empty-dir/f31b5c28-b789-4b2a-a1f6-64d77dffa481-config-out\") pod \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " Oct 30 11:34:55.737944 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.737893 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " Oct 30 11:34:55.737944 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.737936 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f31b5c28-b789-4b2a-a1f6-64d77dffa481-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f31b5c28-b789-4b2a-a1f6-64d77dffa481-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " Oct 30 11:34:55.738225 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.737966 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f31b5c28-b789-4b2a-a1f6-64d77dffa481-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f31b5c28-b789-4b2a-a1f6-64d77dffa481-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " Oct 30 11:34:55.738225 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.738086 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-web-config\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-web-config\") pod \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " Oct 30 11:34:55.738225 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.738162 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f31b5c28-b789-4b2a-a1f6-64d77dffa481-kube-api-access-mgfjk\" (UniqueName: \"kubernetes.io/projected/f31b5c28-b789-4b2a-a1f6-64d77dffa481-kube-api-access-mgfjk\") pod \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " Oct 30 11:34:55.738225 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.738193 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-config\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-config\") pod \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " Oct 30 11:34:55.738480 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.738250 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f31b5c28-b789-4b2a-a1f6-64d77dffa481-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f31b5c28-b789-4b2a-a1f6-64d77dffa481-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " Oct 30 11:34:55.738480 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.738295 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f31b5c28-b789-4b2a-a1f6-64d77dffa481-prometheus-data-science-monitoringstack-rulefiles-1" pod "f31b5c28-b789-4b2a-a1f6-64d77dffa481" (UID: "f31b5c28-b789-4b2a-a1f6-64d77dffa481"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:34:55.738480 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.738302 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " Oct 30 11:34:55.738480 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.738365 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-thanos-prometheus-http-client-file\") pod \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\" (UID: \"f31b5c28-b789-4b2a-a1f6-64d77dffa481\") " Oct 30 11:34:55.738709 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.738670 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f31b5c28-b789-4b2a-a1f6-64d77dffa481-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:55.739036 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.739002 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f31b5c28-b789-4b2a-a1f6-64d77dffa481-prometheus-data-science-monitoringstack-rulefiles-0" pod "f31b5c28-b789-4b2a-a1f6-64d77dffa481" (UID: "f31b5c28-b789-4b2a-a1f6-64d77dffa481"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:34:55.740575 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.740541 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-secret-prometheus-web-tls-ca" pod "f31b5c28-b789-4b2a-a1f6-64d77dffa481" (UID: "f31b5c28-b789-4b2a-a1f6-64d77dffa481"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:55.740688 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.740584 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31b5c28-b789-4b2a-a1f6-64d77dffa481-config-out" pod "f31b5c28-b789-4b2a-a1f6-64d77dffa481" (UID: "f31b5c28-b789-4b2a-a1f6-64d77dffa481"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:34:55.740688 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.740601 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31b5c28-b789-4b2a-a1f6-64d77dffa481-tls-assets" pod "f31b5c28-b789-4b2a-a1f6-64d77dffa481" (UID: "f31b5c28-b789-4b2a-a1f6-64d77dffa481"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:34:55.740965 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.740928 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f31b5c28-b789-4b2a-a1f6-64d77dffa481-prometheus-data-science-monitoringstack-rulefiles-2" pod "f31b5c28-b789-4b2a-a1f6-64d77dffa481" (UID: "f31b5c28-b789-4b2a-a1f6-64d77dffa481"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:34:55.741148 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.741123 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "f31b5c28-b789-4b2a-a1f6-64d77dffa481" (UID: "f31b5c28-b789-4b2a-a1f6-64d77dffa481"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:55.741745 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.741725 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-thanos-prometheus-http-client-file" pod "f31b5c28-b789-4b2a-a1f6-64d77dffa481" (UID: "f31b5c28-b789-4b2a-a1f6-64d77dffa481"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:55.741843 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.741749 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31b5c28-b789-4b2a-a1f6-64d77dffa481-kube-api-access-mgfjk" pod "f31b5c28-b789-4b2a-a1f6-64d77dffa481" (UID: "f31b5c28-b789-4b2a-a1f6-64d77dffa481"). InnerVolumeSpecName "kube-api-access-mgfjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:34:55.741843 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.741754 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "f31b5c28-b789-4b2a-a1f6-64d77dffa481" (UID: "f31b5c28-b789-4b2a-a1f6-64d77dffa481"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:55.742885 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.742864 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-config" pod "f31b5c28-b789-4b2a-a1f6-64d77dffa481" (UID: "f31b5c28-b789-4b2a-a1f6-64d77dffa481"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:55.743729 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.743712 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48" pod "f31b5c28-b789-4b2a-a1f6-64d77dffa481" (UID: "f31b5c28-b789-4b2a-a1f6-64d77dffa481"). InnerVolumeSpecName "pvc-c799cc48-125a-421c-8fc6-8a4d017a7653". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 11:34:55.753462 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.753404 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-web-config" pod "f31b5c28-b789-4b2a-a1f6-64d77dffa481" (UID: "f31b5c28-b789-4b2a-a1f6-64d77dffa481"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:34:55.839441 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.839404 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f31b5c28-b789-4b2a-a1f6-64d77dffa481-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:55.839619 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.839468 4289 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:55.839619 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.839489 4289 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:55.839619 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.839529 4289 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") on node \"ip-10-0-137-242.ec2.internal\" " Oct 30 11:34:55.839619 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.839541 4289 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-secret-prometheus-web-tls-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:55.839619 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.839550 4289 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f31b5c28-b789-4b2a-a1f6-64d77dffa481-tls-assets\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:55.839619 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.839559 4289 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f31b5c28-b789-4b2a-a1f6-64d77dffa481-config-out\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:55.839619 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.839567 4289 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:55.839619 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.839576 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f31b5c28-b789-4b2a-a1f6-64d77dffa481-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:55.839619 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.839584 4289 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-web-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:55.839619 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.839593 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgfjk\" (UniqueName: \"kubernetes.io/projected/f31b5c28-b789-4b2a-a1f6-64d77dffa481-kube-api-access-mgfjk\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:55.839619 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.839600 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f31b5c28-b789-4b2a-a1f6-64d77dffa481-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:34:55.849285 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.849266 4289 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-c799cc48-125a-421c-8fc6-8a4d017a7653" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48") on node "ip-10-0-137-242.ec2.internal" Oct 30 11:34:55.940378 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:55.940345 4289 reconciler_common.go:299] "Volume detached for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"csi-ea06c66241a0a003a5daff5d9fc602f40e53768b00b104c5e4118b92d558c090\"" Oct 30 11:34:56.105965 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:56.105880 4289 generic.go:356] "Generic (PLEG): container finished" podID="f31b5c28-b789-4b2a-a1f6-64d77dffa481" containerID="36ec3031da98de911ccd0c0b74981aeeb4673cc0af6c46cdc4367d874530ce8f" exitCode=0 Oct 30 11:34:56.106120 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:56.105962 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:34:56.106120 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:56.105968 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f31b5c28-b789-4b2a-a1f6-64d77dffa481","Type":"ContainerDied","Data":"36ec3031da98de911ccd0c0b74981aeeb4673cc0af6c46cdc4367d874530ce8f"} Oct 30 11:34:56.106120 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:56.106065 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f31b5c28-b789-4b2a-a1f6-64d77dffa481","Type":"ContainerDied","Data":"a87a0754589140645a6962d914b751d2af594708537ea6c9ce011876a185e28c"} Oct 30 11:34:56.106120 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:56.106099 4289 scope.go:122] "RemoveContainer" containerID="78fa504f2cea0ded06c3699d3a6cb64372eb90e1f27d7fd1f324ee9ca1db8ec1" Oct 30 11:34:56.114799 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:56.114780 4289 scope.go:122] "RemoveContainer" containerID="df76779925e290ac771ceba64e602d0dc83eb576effab05870852f41c741d61e" Oct 30 11:34:56.122668 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:56.122651 4289 scope.go:122] "RemoveContainer" containerID="36ec3031da98de911ccd0c0b74981aeeb4673cc0af6c46cdc4367d874530ce8f" Oct 30 11:34:56.130205 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:56.130177 4289 scope.go:122] "RemoveContainer" containerID="cc4f83084404b18077aa2eff2447531865ba015c7fd17c0c77f2310f43fdbc6d" Oct 30 11:34:56.137555 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:56.137538 4289 scope.go:122] "RemoveContainer" containerID="78fa504f2cea0ded06c3699d3a6cb64372eb90e1f27d7fd1f324ee9ca1db8ec1" Oct 30 11:34:56.137807 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:56.137787 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78fa504f2cea0ded06c3699d3a6cb64372eb90e1f27d7fd1f324ee9ca1db8ec1\": container with ID starting with 78fa504f2cea0ded06c3699d3a6cb64372eb90e1f27d7fd1f324ee9ca1db8ec1 not found: ID does not exist" containerID="78fa504f2cea0ded06c3699d3a6cb64372eb90e1f27d7fd1f324ee9ca1db8ec1" Oct 30 11:34:56.137907 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:56.137814 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78fa504f2cea0ded06c3699d3a6cb64372eb90e1f27d7fd1f324ee9ca1db8ec1"} err="failed to get container status \"78fa504f2cea0ded06c3699d3a6cb64372eb90e1f27d7fd1f324ee9ca1db8ec1\": rpc error: code = NotFound desc = could not find container \"78fa504f2cea0ded06c3699d3a6cb64372eb90e1f27d7fd1f324ee9ca1db8ec1\": container with ID starting with 78fa504f2cea0ded06c3699d3a6cb64372eb90e1f27d7fd1f324ee9ca1db8ec1 not found: ID does not exist" Oct 30 11:34:56.137907 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:56.137828 4289 scope.go:122] "RemoveContainer" containerID="df76779925e290ac771ceba64e602d0dc83eb576effab05870852f41c741d61e" Oct 30 11:34:56.138065 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:56.138046 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df76779925e290ac771ceba64e602d0dc83eb576effab05870852f41c741d61e\": container with ID starting with df76779925e290ac771ceba64e602d0dc83eb576effab05870852f41c741d61e not found: ID does not exist" containerID="df76779925e290ac771ceba64e602d0dc83eb576effab05870852f41c741d61e" Oct 30 11:34:56.138104 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:56.138072 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df76779925e290ac771ceba64e602d0dc83eb576effab05870852f41c741d61e"} err="failed to get container status \"df76779925e290ac771ceba64e602d0dc83eb576effab05870852f41c741d61e\": rpc error: code = NotFound desc = could not find container \"df76779925e290ac771ceba64e602d0dc83eb576effab05870852f41c741d61e\": container with ID starting with df76779925e290ac771ceba64e602d0dc83eb576effab05870852f41c741d61e not found: ID does not exist" Oct 30 11:34:56.138104 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:56.138089 4289 scope.go:122] "RemoveContainer" containerID="36ec3031da98de911ccd0c0b74981aeeb4673cc0af6c46cdc4367d874530ce8f" Oct 30 11:34:56.138301 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:56.138287 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36ec3031da98de911ccd0c0b74981aeeb4673cc0af6c46cdc4367d874530ce8f\": container with ID starting with 36ec3031da98de911ccd0c0b74981aeeb4673cc0af6c46cdc4367d874530ce8f not found: ID does not exist" containerID="36ec3031da98de911ccd0c0b74981aeeb4673cc0af6c46cdc4367d874530ce8f" Oct 30 11:34:56.138347 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:56.138305 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36ec3031da98de911ccd0c0b74981aeeb4673cc0af6c46cdc4367d874530ce8f"} err="failed to get container status \"36ec3031da98de911ccd0c0b74981aeeb4673cc0af6c46cdc4367d874530ce8f\": rpc error: code = NotFound desc = could not find container \"36ec3031da98de911ccd0c0b74981aeeb4673cc0af6c46cdc4367d874530ce8f\": container with ID starting with 36ec3031da98de911ccd0c0b74981aeeb4673cc0af6c46cdc4367d874530ce8f not found: ID does not exist" Oct 30 11:34:56.138347 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:56.138320 4289 scope.go:122] "RemoveContainer" containerID="cc4f83084404b18077aa2eff2447531865ba015c7fd17c0c77f2310f43fdbc6d" Oct 30 11:34:56.138542 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:56.138526 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc4f83084404b18077aa2eff2447531865ba015c7fd17c0c77f2310f43fdbc6d\": container with ID starting with cc4f83084404b18077aa2eff2447531865ba015c7fd17c0c77f2310f43fdbc6d not found: ID does not exist" containerID="cc4f83084404b18077aa2eff2447531865ba015c7fd17c0c77f2310f43fdbc6d" Oct 30 11:34:56.138598 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:56.138545 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc4f83084404b18077aa2eff2447531865ba015c7fd17c0c77f2310f43fdbc6d"} err="failed to get container status \"cc4f83084404b18077aa2eff2447531865ba015c7fd17c0c77f2310f43fdbc6d\": rpc error: code = NotFound desc = could not find container \"cc4f83084404b18077aa2eff2447531865ba015c7fd17c0c77f2310f43fdbc6d\": container with ID starting with cc4f83084404b18077aa2eff2447531865ba015c7fd17c0c77f2310f43fdbc6d not found: ID does not exist" Oct 30 11:34:56.159293 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:56.159270 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:34:56.171859 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:56.171829 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:34:56.257619 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:34:56.257587 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:34:57.624246 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:57.624212 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f31b5c28-b789-4b2a-a1f6-64d77dffa481" path="/var/lib/kubelet/pods/f31b5c28-b789-4b2a-a1f6-64d77dffa481/volumes" Oct 30 11:34:59.232112 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:34:59.232082 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:00.164101 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.164071 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:00.330119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.330086 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:00.395015 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.394983 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6"] Oct 30 11:35:00.395291 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395279 4289 cpu_manager.go:414] "containerMap: removing container" podUID="b029b579-df40-408b-a55b-34a9ac9a1f12" containerName="korrel8r" Oct 30 11:35:00.395338 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395292 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b029b579-df40-408b-a55b-34a9ac9a1f12" containerName="korrel8r" Oct 30 11:35:00.395338 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395303 4289 cpu_manager.go:414] "containerMap: removing container" podUID="c34bf4cd-d232-45d9-8e12-40ea4859b874" containerName="config-reloader" Oct 30 11:35:00.395338 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395308 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34bf4cd-d232-45d9-8e12-40ea4859b874" containerName="config-reloader" Oct 30 11:35:00.395338 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395318 4289 cpu_manager.go:414] "containerMap: removing container" podUID="56f0158b-1fcb-4417-8182-995b72813052" containerName="prom-label-proxy" Oct 30 11:35:00.395338 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395324 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f0158b-1fcb-4417-8182-995b72813052" containerName="prom-label-proxy" Oct 30 11:35:00.395338 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395330 4289 cpu_manager.go:414] "containerMap: removing container" podUID="34d3af29-0917-4d1d-934d-fbab9aa8f38b" containerName="prom-label-proxy" Oct 30 11:35:00.395338 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395335 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d3af29-0917-4d1d-934d-fbab9aa8f38b" containerName="prom-label-proxy" Oct 30 11:35:00.395338 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395341 4289 cpu_manager.go:414] "containerMap: removing container" podUID="c34bf4cd-d232-45d9-8e12-40ea4859b874" containerName="init-config-reloader" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395346 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34bf4cd-d232-45d9-8e12-40ea4859b874" containerName="init-config-reloader" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395353 4289 cpu_manager.go:414] "containerMap: removing container" podUID="f31b5c28-b789-4b2a-a1f6-64d77dffa481" containerName="config-reloader" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395358 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f31b5c28-b789-4b2a-a1f6-64d77dffa481" containerName="config-reloader" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395368 4289 cpu_manager.go:414] "containerMap: removing container" podUID="1808eff5-6fef-47ae-b389-2f8e0c4859c4" containerName="otc-container" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395374 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1808eff5-6fef-47ae-b389-2f8e0c4859c4" containerName="otc-container" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395384 4289 cpu_manager.go:414] "containerMap: removing container" podUID="56f0158b-1fcb-4417-8182-995b72813052" containerName="method-gate" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395388 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f0158b-1fcb-4417-8182-995b72813052" containerName="method-gate" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395395 4289 cpu_manager.go:414] "containerMap: removing container" podUID="c34bf4cd-d232-45d9-8e12-40ea4859b874" containerName="alertmanager" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395399 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34bf4cd-d232-45d9-8e12-40ea4859b874" containerName="alertmanager" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395406 4289 cpu_manager.go:414] "containerMap: removing container" podUID="34d3af29-0917-4d1d-934d-fbab9aa8f38b" containerName="method-gate" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395410 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d3af29-0917-4d1d-934d-fbab9aa8f38b" containerName="method-gate" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395415 4289 cpu_manager.go:414] "containerMap: removing container" podUID="e79c96e2-ffe5-41d4-823c-b86b416aa4f8" containerName="kube-rbac-proxy" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395420 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e79c96e2-ffe5-41d4-823c-b86b416aa4f8" containerName="kube-rbac-proxy" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395425 4289 cpu_manager.go:414] "containerMap: removing container" podUID="f31b5c28-b789-4b2a-a1f6-64d77dffa481" containerName="prometheus" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395442 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f31b5c28-b789-4b2a-a1f6-64d77dffa481" containerName="prometheus" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395452 4289 cpu_manager.go:414] "containerMap: removing container" podUID="34d3af29-0917-4d1d-934d-fbab9aa8f38b" containerName="kube-rbac-proxy" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395456 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d3af29-0917-4d1d-934d-fbab9aa8f38b" containerName="kube-rbac-proxy" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395465 4289 cpu_manager.go:414] "containerMap: removing container" podUID="f31b5c28-b789-4b2a-a1f6-64d77dffa481" containerName="init-config-reloader" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395477 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f31b5c28-b789-4b2a-a1f6-64d77dffa481" containerName="init-config-reloader" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395482 4289 cpu_manager.go:414] "containerMap: removing container" podUID="f31b5c28-b789-4b2a-a1f6-64d77dffa481" containerName="thanos-sidecar" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395486 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f31b5c28-b789-4b2a-a1f6-64d77dffa481" containerName="thanos-sidecar" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395497 4289 cpu_manager.go:414] "containerMap: removing container" podUID="56f0158b-1fcb-4417-8182-995b72813052" containerName="kube-rbac-proxy" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395501 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f0158b-1fcb-4417-8182-995b72813052" containerName="kube-rbac-proxy" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395556 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="34d3af29-0917-4d1d-934d-fbab9aa8f38b" containerName="kube-rbac-proxy" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395565 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="f31b5c28-b789-4b2a-a1f6-64d77dffa481" containerName="thanos-sidecar" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395572 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f0158b-1fcb-4417-8182-995b72813052" containerName="kube-rbac-proxy" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395578 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="b029b579-df40-408b-a55b-34a9ac9a1f12" containerName="korrel8r" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395584 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34bf4cd-d232-45d9-8e12-40ea4859b874" containerName="config-reloader" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395591 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f0158b-1fcb-4417-8182-995b72813052" containerName="prom-label-proxy" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395595 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="34d3af29-0917-4d1d-934d-fbab9aa8f38b" containerName="prom-label-proxy" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395600 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="f31b5c28-b789-4b2a-a1f6-64d77dffa481" containerName="config-reloader" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395607 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="1808eff5-6fef-47ae-b389-2f8e0c4859c4" containerName="otc-container" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395612 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f0158b-1fcb-4417-8182-995b72813052" containerName="method-gate" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395617 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34bf4cd-d232-45d9-8e12-40ea4859b874" containerName="alertmanager" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395621 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="34d3af29-0917-4d1d-934d-fbab9aa8f38b" containerName="method-gate" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395626 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="e79c96e2-ffe5-41d4-823c-b86b416aa4f8" containerName="kube-rbac-proxy" Oct 30 11:35:00.395651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.395631 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="f31b5c28-b789-4b2a-a1f6-64d77dffa481" containerName="prometheus" Oct 30 11:35:00.398336 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.398321 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" Oct 30 11:35:00.427995 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.427925 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 11:35:00.427995 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.427925 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 11:35:00.428156 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.427989 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 11:35:00.428416 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.428398 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 11:35:00.429413 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.429390 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6"] Oct 30 11:35:00.431260 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.431238 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 11:35:00.431260 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.431253 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 11:35:00.431401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.431328 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-jmnrf\"" Oct 30 11:35:00.442575 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.442547 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd"] Oct 30 11:35:00.446438 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.446413 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" Oct 30 11:35:00.455663 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.455633 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd"] Oct 30 11:35:00.476622 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.476593 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz"] Oct 30 11:35:00.483670 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.481492 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" Oct 30 11:35:00.483670 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.483043 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcwrt\" (UniqueName: \"kubernetes.io/projected/c1a60678-72d4-453c-8c02-ea030952c1ce-kube-api-access-kcwrt\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-gnxk6\" (UID: \"c1a60678-72d4-453c-8c02-ea030952c1ce\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" Oct 30 11:35:00.483670 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.483113 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-gnxk6\" (UID: \"c1a60678-72d4-453c-8c02-ea030952c1ce\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" Oct 30 11:35:00.483670 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.483162 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/c1a60678-72d4-453c-8c02-ea030952c1ce-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-gnxk6\" (UID: \"c1a60678-72d4-453c-8c02-ea030952c1ce\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" Oct 30 11:35:00.483670 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.483214 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-gnxk6\" (UID: \"c1a60678-72d4-453c-8c02-ea030952c1ce\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" Oct 30 11:35:00.483670 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.483241 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-gnxk6\" (UID: \"c1a60678-72d4-453c-8c02-ea030952c1ce\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" Oct 30 11:35:00.499968 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.499938 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz"] Oct 30 11:35:00.513062 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.513028 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:00.514909 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.514888 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz"] Oct 30 11:35:00.518133 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.518118 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz" Oct 30 11:35:00.519187 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.519165 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 11:35:00.519278 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.519210 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 11:35:00.519278 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.519236 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 11:35:00.519692 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.519678 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 11:35:00.519737 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.519701 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-7fpmr\"" Oct 30 11:35:00.519772 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.519743 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 11:35:00.519875 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.519856 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-cx4lb\"" Oct 30 11:35:00.536916 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.536891 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz"] Oct 30 11:35:00.583854 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.583818 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pq24\" (UniqueName: \"kubernetes.io/projected/c23b2839-9e2a-4465-8825-ab1a1d82dada-kube-api-access-2pq24\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-79scd\" (UID: \"c23b2839-9e2a-4465-8825-ab1a1d82dada\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" Oct 30 11:35:00.584045 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.583875 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-gnxk6\" (UID: \"c1a60678-72d4-453c-8c02-ea030952c1ce\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" Oct 30 11:35:00.584045 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.583899 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c23b2839-9e2a-4465-8825-ab1a1d82dada-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-79scd\" (UID: \"c23b2839-9e2a-4465-8825-ab1a1d82dada\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" Oct 30 11:35:00.584178 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.584099 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ff9592ba-475a-4ced-b3a2-92694daf0e15-tls\") pod \"korrel8r-5cbcc89b8f-gj5lz\" (UID: \"ff9592ba-475a-4ced-b3a2-92694daf0e15\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz" Oct 30 11:35:00.584178 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.584129 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqms5\" (UniqueName: \"kubernetes.io/projected/ff9592ba-475a-4ced-b3a2-92694daf0e15-kube-api-access-nqms5\") pod \"korrel8r-5cbcc89b8f-gj5lz\" (UID: \"ff9592ba-475a-4ced-b3a2-92694daf0e15\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz" Oct 30 11:35:00.584178 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.584155 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/c1a60678-72d4-453c-8c02-ea030952c1ce-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-gnxk6\" (UID: \"c1a60678-72d4-453c-8c02-ea030952c1ce\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" Oct 30 11:35:00.584349 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.584225 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/708f844f-1c03-4e6f-95b9-7f3b8dabff06-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-kztxz\" (UID: \"708f844f-1c03-4e6f-95b9-7f3b8dabff06\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" Oct 30 11:35:00.584349 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.584259 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-gnxk6\" (UID: \"c1a60678-72d4-453c-8c02-ea030952c1ce\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" Oct 30 11:35:00.584349 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.584275 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-gnxk6\" (UID: \"c1a60678-72d4-453c-8c02-ea030952c1ce\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" Oct 30 11:35:00.584349 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.584289 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff9592ba-475a-4ced-b3a2-92694daf0e15-config\") pod \"korrel8r-5cbcc89b8f-gj5lz\" (UID: \"ff9592ba-475a-4ced-b3a2-92694daf0e15\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz" Oct 30 11:35:00.584556 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.584348 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c23b2839-9e2a-4465-8825-ab1a1d82dada-client-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-79scd\" (UID: \"c23b2839-9e2a-4465-8825-ab1a1d82dada\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" Oct 30 11:35:00.584556 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.584378 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/708f844f-1c03-4e6f-95b9-7f3b8dabff06-config\") pod \"data-science-thanos-querier-proxy-7b56895dd5-kztxz\" (UID: \"708f844f-1c03-4e6f-95b9-7f3b8dabff06\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" Oct 30 11:35:00.584556 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:35:00.584479 4289 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 30 11:35:00.584781 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:35:00.584569 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-prometheus-client-cert podName:c1a60678-72d4-453c-8c02-ea030952c1ce nodeName:}" failed. No retries permitted until 2026-10-30 11:35:01.084545709 +0000 UTC m=+2991.585925174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" (UID: "c1a60678-72d4-453c-8c02-ea030952c1ce") : secret "prometheus-operated-tls" not found Oct 30 11:35:00.584781 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.584618 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcwrt\" (UniqueName: \"kubernetes.io/projected/c1a60678-72d4-453c-8c02-ea030952c1ce-kube-api-access-kcwrt\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-gnxk6\" (UID: \"c1a60678-72d4-453c-8c02-ea030952c1ce\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" Oct 30 11:35:00.584781 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.584646 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c23b2839-9e2a-4465-8825-ab1a1d82dada-config\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-79scd\" (UID: \"c23b2839-9e2a-4465-8825-ab1a1d82dada\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" Oct 30 11:35:00.584781 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.584670 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/c23b2839-9e2a-4465-8825-ab1a1d82dada-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-79scd\" (UID: \"c23b2839-9e2a-4465-8825-ab1a1d82dada\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" Oct 30 11:35:00.584781 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.584697 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz89t\" (UniqueName: \"kubernetes.io/projected/708f844f-1c03-4e6f-95b9-7f3b8dabff06-kube-api-access-sz89t\") pod \"data-science-thanos-querier-proxy-7b56895dd5-kztxz\" (UID: \"708f844f-1c03-4e6f-95b9-7f3b8dabff06\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" Oct 30 11:35:00.585070 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.584854 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/c1a60678-72d4-453c-8c02-ea030952c1ce-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-gnxk6\" (UID: \"c1a60678-72d4-453c-8c02-ea030952c1ce\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" Oct 30 11:35:00.586439 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.586405 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-gnxk6\" (UID: \"c1a60678-72d4-453c-8c02-ea030952c1ce\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" Oct 30 11:35:00.586595 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.586579 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-gnxk6\" (UID: \"c1a60678-72d4-453c-8c02-ea030952c1ce\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" Oct 30 11:35:00.608066 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.608038 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcwrt\" (UniqueName: \"kubernetes.io/projected/c1a60678-72d4-453c-8c02-ea030952c1ce-kube-api-access-kcwrt\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-gnxk6\" (UID: \"c1a60678-72d4-453c-8c02-ea030952c1ce\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" Oct 30 11:35:00.619565 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.619541 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 11:35:00.619855 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.619829 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-wtrpk\"" Oct 30 11:35:00.619931 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.619877 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 11:35:00.681530 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.681454 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:00.685630 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.685599 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c23b2839-9e2a-4465-8825-ab1a1d82dada-client-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-79scd\" (UID: \"c23b2839-9e2a-4465-8825-ab1a1d82dada\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" Oct 30 11:35:00.685766 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.685639 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/708f844f-1c03-4e6f-95b9-7f3b8dabff06-config\") pod \"data-science-thanos-querier-proxy-7b56895dd5-kztxz\" (UID: \"708f844f-1c03-4e6f-95b9-7f3b8dabff06\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" Oct 30 11:35:00.685766 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.685690 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c23b2839-9e2a-4465-8825-ab1a1d82dada-config\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-79scd\" (UID: \"c23b2839-9e2a-4465-8825-ab1a1d82dada\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" Oct 30 11:35:00.685766 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.685713 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/c23b2839-9e2a-4465-8825-ab1a1d82dada-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-79scd\" (UID: \"c23b2839-9e2a-4465-8825-ab1a1d82dada\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" Oct 30 11:35:00.685766 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.685739 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz89t\" (UniqueName: \"kubernetes.io/projected/708f844f-1c03-4e6f-95b9-7f3b8dabff06-kube-api-access-sz89t\") pod \"data-science-thanos-querier-proxy-7b56895dd5-kztxz\" (UID: \"708f844f-1c03-4e6f-95b9-7f3b8dabff06\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" Oct 30 11:35:00.685989 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.685773 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pq24\" (UniqueName: \"kubernetes.io/projected/c23b2839-9e2a-4465-8825-ab1a1d82dada-kube-api-access-2pq24\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-79scd\" (UID: \"c23b2839-9e2a-4465-8825-ab1a1d82dada\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" Oct 30 11:35:00.685989 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.685835 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c23b2839-9e2a-4465-8825-ab1a1d82dada-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-79scd\" (UID: \"c23b2839-9e2a-4465-8825-ab1a1d82dada\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" Oct 30 11:35:00.685989 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.685858 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ff9592ba-475a-4ced-b3a2-92694daf0e15-tls\") pod \"korrel8r-5cbcc89b8f-gj5lz\" (UID: \"ff9592ba-475a-4ced-b3a2-92694daf0e15\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz" Oct 30 11:35:00.685989 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.685884 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqms5\" (UniqueName: \"kubernetes.io/projected/ff9592ba-475a-4ced-b3a2-92694daf0e15-kube-api-access-nqms5\") pod \"korrel8r-5cbcc89b8f-gj5lz\" (UID: \"ff9592ba-475a-4ced-b3a2-92694daf0e15\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz" Oct 30 11:35:00.685989 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.685929 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/708f844f-1c03-4e6f-95b9-7f3b8dabff06-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-kztxz\" (UID: \"708f844f-1c03-4e6f-95b9-7f3b8dabff06\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" Oct 30 11:35:00.686245 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.685998 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff9592ba-475a-4ced-b3a2-92694daf0e15-config\") pod \"korrel8r-5cbcc89b8f-gj5lz\" (UID: \"ff9592ba-475a-4ced-b3a2-92694daf0e15\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz" Oct 30 11:35:00.686354 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.686327 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c23b2839-9e2a-4465-8825-ab1a1d82dada-client-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-79scd\" (UID: \"c23b2839-9e2a-4465-8825-ab1a1d82dada\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" Oct 30 11:35:00.686454 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:35:00.686331 4289 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-thanos-querier-proxy-tls: secret "data-science-thanos-querier-proxy-tls" not found Oct 30 11:35:00.686454 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.686386 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/708f844f-1c03-4e6f-95b9-7f3b8dabff06-config\") pod \"data-science-thanos-querier-proxy-7b56895dd5-kztxz\" (UID: \"708f844f-1c03-4e6f-95b9-7f3b8dabff06\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" Oct 30 11:35:00.686454 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:35:00.686424 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/708f844f-1c03-4e6f-95b9-7f3b8dabff06-tls podName:708f844f-1c03-4e6f-95b9-7f3b8dabff06 nodeName:}" failed. No retries permitted until 2026-10-30 11:35:01.186405522 +0000 UTC m=+2991.687785000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/708f844f-1c03-4e6f-95b9-7f3b8dabff06-tls") pod "data-science-thanos-querier-proxy-7b56895dd5-kztxz" (UID: "708f844f-1c03-4e6f-95b9-7f3b8dabff06") : secret "data-science-thanos-querier-proxy-tls" not found Oct 30 11:35:00.686650 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.686507 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/c23b2839-9e2a-4465-8825-ab1a1d82dada-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-79scd\" (UID: \"c23b2839-9e2a-4465-8825-ab1a1d82dada\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" Oct 30 11:35:00.686650 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.686529 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c23b2839-9e2a-4465-8825-ab1a1d82dada-config\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-79scd\" (UID: \"c23b2839-9e2a-4465-8825-ab1a1d82dada\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" Oct 30 11:35:00.687117 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.687096 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff9592ba-475a-4ced-b3a2-92694daf0e15-config\") pod \"korrel8r-5cbcc89b8f-gj5lz\" (UID: \"ff9592ba-475a-4ced-b3a2-92694daf0e15\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz" Oct 30 11:35:00.688831 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.688808 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c23b2839-9e2a-4465-8825-ab1a1d82dada-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-79scd\" (UID: \"c23b2839-9e2a-4465-8825-ab1a1d82dada\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" Oct 30 11:35:00.688918 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.688840 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ff9592ba-475a-4ced-b3a2-92694daf0e15-tls\") pod \"korrel8r-5cbcc89b8f-gj5lz\" (UID: \"ff9592ba-475a-4ced-b3a2-92694daf0e15\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz" Oct 30 11:35:00.710061 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.710037 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqms5\" (UniqueName: \"kubernetes.io/projected/ff9592ba-475a-4ced-b3a2-92694daf0e15-kube-api-access-nqms5\") pod \"korrel8r-5cbcc89b8f-gj5lz\" (UID: \"ff9592ba-475a-4ced-b3a2-92694daf0e15\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz" Oct 30 11:35:00.710061 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.710056 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pq24\" (UniqueName: \"kubernetes.io/projected/c23b2839-9e2a-4465-8825-ab1a1d82dada-kube-api-access-2pq24\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-79scd\" (UID: \"c23b2839-9e2a-4465-8825-ab1a1d82dada\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" Oct 30 11:35:00.710190 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.710103 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz89t\" (UniqueName: \"kubernetes.io/projected/708f844f-1c03-4e6f-95b9-7f3b8dabff06-kube-api-access-sz89t\") pod \"data-science-thanos-querier-proxy-7b56895dd5-kztxz\" (UID: \"708f844f-1c03-4e6f-95b9-7f3b8dabff06\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" Oct 30 11:35:00.764861 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.764833 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" Oct 30 11:35:00.831121 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.831094 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz" Oct 30 11:35:00.856956 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:00.856929 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:01.039915 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:01.039880 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:01.075708 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:01.075684 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd"] Oct 30 11:35:01.077453 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:35:01.077407 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc23b2839_9e2a_4465_8825_ab1a1d82dada.slice/crio-8c78399eb93437b784ef653ae1ca3e197f5f577594ba1b77f6f5e167fc54cdf5 WatchSource:0}: Error finding container 8c78399eb93437b784ef653ae1ca3e197f5f577594ba1b77f6f5e167fc54cdf5: Status 404 returned error can't find the container with id 8c78399eb93437b784ef653ae1ca3e197f5f577594ba1b77f6f5e167fc54cdf5 Oct 30 11:35:01.090296 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:01.090276 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-gnxk6\" (UID: \"c1a60678-72d4-453c-8c02-ea030952c1ce\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" Oct 30 11:35:01.090410 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:35:01.090397 4289 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 30 11:35:01.090471 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:35:01.090462 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-prometheus-client-cert podName:c1a60678-72d4-453c-8c02-ea030952c1ce nodeName:}" failed. No retries permitted until 2026-10-30 11:35:02.090448474 +0000 UTC m=+2992.591827958 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" (UID: "c1a60678-72d4-453c-8c02-ea030952c1ce") : secret "prometheus-operated-tls" not found Oct 30 11:35:01.131379 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:01.131254 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" event={"ID":"c23b2839-9e2a-4465-8825-ab1a1d82dada","Type":"ContainerStarted","Data":"8c78399eb93437b784ef653ae1ca3e197f5f577594ba1b77f6f5e167fc54cdf5"} Oct 30 11:35:01.131779 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:01.131735 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz"] Oct 30 11:35:01.133822 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:35:01.133799 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff9592ba_475a_4ced_b3a2_92694daf0e15.slice/crio-7570a613aa15af2a1d59bf425557655020c0c17bce49af2a99a4140600e4df82 WatchSource:0}: Error finding container 7570a613aa15af2a1d59bf425557655020c0c17bce49af2a99a4140600e4df82: Status 404 returned error can't find the container with id 7570a613aa15af2a1d59bf425557655020c0c17bce49af2a99a4140600e4df82 Oct 30 11:35:01.191567 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:01.191541 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/708f844f-1c03-4e6f-95b9-7f3b8dabff06-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-kztxz\" (UID: \"708f844f-1c03-4e6f-95b9-7f3b8dabff06\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" Oct 30 11:35:01.193878 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:01.193855 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/708f844f-1c03-4e6f-95b9-7f3b8dabff06-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-kztxz\" (UID: \"708f844f-1c03-4e6f-95b9-7f3b8dabff06\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" Oct 30 11:35:01.401367 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:01.401285 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" Oct 30 11:35:01.410197 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:01.410174 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:01.608684 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:01.608647 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:01.709251 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:01.709226 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz"] Oct 30 11:35:01.711858 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:35:01.711823 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708f844f_1c03_4e6f_95b9_7f3b8dabff06.slice/crio-71dda2a28a3783017de09368a57d579076cca788dfadd272b312986914a8d303 WatchSource:0}: Error finding container 71dda2a28a3783017de09368a57d579076cca788dfadd272b312986914a8d303: Status 404 returned error can't find the container with id 71dda2a28a3783017de09368a57d579076cca788dfadd272b312986914a8d303 Oct 30 11:35:02.058001 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:02.057973 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:02.100882 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:02.100847 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-gnxk6\" (UID: \"c1a60678-72d4-453c-8c02-ea030952c1ce\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" Oct 30 11:35:02.103323 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:02.103300 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-gnxk6\" (UID: \"c1a60678-72d4-453c-8c02-ea030952c1ce\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" Oct 30 11:35:02.139742 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:02.139709 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" event={"ID":"708f844f-1c03-4e6f-95b9-7f3b8dabff06","Type":"ContainerStarted","Data":"bf7e5ead5ca9a3126391e7bba5b706e9c922675b9aec21a900e9a0633c26ba64"} Oct 30 11:35:02.139913 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:02.139778 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" event={"ID":"708f844f-1c03-4e6f-95b9-7f3b8dabff06","Type":"ContainerStarted","Data":"24698e8db6f2a06f965ed6cbcc4dfd67ef3c0ad979c3a0b304d243665277957e"} Oct 30 11:35:02.139913 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:02.139817 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" event={"ID":"708f844f-1c03-4e6f-95b9-7f3b8dabff06","Type":"ContainerStarted","Data":"ec4ea8bc43eef93b09798857da7657d14e597d594cfcfcff6f7d3b515033bd43"} Oct 30 11:35:02.139913 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:02.139858 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" event={"ID":"708f844f-1c03-4e6f-95b9-7f3b8dabff06","Type":"ContainerStarted","Data":"71dda2a28a3783017de09368a57d579076cca788dfadd272b312986914a8d303"} Oct 30 11:35:02.141508 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:02.141476 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz" event={"ID":"ff9592ba-475a-4ced-b3a2-92694daf0e15","Type":"ContainerStarted","Data":"428f96ce2cee38489f5882fb90d1a118956a227929f67ee5291ee9c3e3bb1805"} Oct 30 11:35:02.141646 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:02.141562 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz" Oct 30 11:35:02.141646 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:02.141592 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz" event={"ID":"ff9592ba-475a-4ced-b3a2-92694daf0e15","Type":"ContainerStarted","Data":"7570a613aa15af2a1d59bf425557655020c0c17bce49af2a99a4140600e4df82"} Oct 30 11:35:02.143891 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:02.143863 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" event={"ID":"c23b2839-9e2a-4465-8825-ab1a1d82dada","Type":"ContainerStarted","Data":"19e0f03fa4e00667e1cca4bf102c8abb1ca96f9b0e5b1677f8adac76e4954ec6"} Oct 30 11:35:02.143986 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:02.143935 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" event={"ID":"c23b2839-9e2a-4465-8825-ab1a1d82dada","Type":"ContainerStarted","Data":"a3dd94f4c2a4581caf4c608afc5aad0fe0f7c8c5b9fc0680a59572c7f18aec2c"} Oct 30 11:35:02.143986 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:02.143976 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" event={"ID":"c23b2839-9e2a-4465-8825-ab1a1d82dada","Type":"ContainerStarted","Data":"00276833e29815457b8fdea7aaa0d696f658648f35ffee255d8f7e9becbf8906"} Oct 30 11:35:02.211502 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:02.211463 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" Oct 30 11:35:02.227514 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:02.227421 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" podStartSLOduration=2.227403332 podStartE2EDuration="2.227403332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:02.184625927 +0000 UTC m=+2992.686005416" watchObservedRunningTime="2026-10-30 11:35:02.227403332 +0000 UTC m=+2992.728782818" Oct 30 11:35:02.227668 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:02.227648 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz" podStartSLOduration=2.227639724 podStartE2EDuration="2.227639724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:02.226733417 +0000 UTC m=+2992.728112902" watchObservedRunningTime="2026-10-30 11:35:02.227639724 +0000 UTC m=+2992.729019208" Oct 30 11:35:02.232681 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:02.232653 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:02.272299 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:02.272238 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" podStartSLOduration=2.272218977 podStartE2EDuration="2.272218977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:02.271482524 +0000 UTC m=+2992.772862011" watchObservedRunningTime="2026-10-30 11:35:02.272218977 +0000 UTC m=+2992.773598463" Oct 30 11:35:02.508859 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:02.508781 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6"] Oct 30 11:35:02.511122 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:35:02.511096 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a60678_72d4_453c_8c02_ea030952c1ce.slice/crio-72842edaeb90b7e794b13e27f439fcf293c6a496091ff2167722018c910e74e8 WatchSource:0}: Error finding container 72842edaeb90b7e794b13e27f439fcf293c6a496091ff2167722018c910e74e8: Status 404 returned error can't find the container with id 72842edaeb90b7e794b13e27f439fcf293c6a496091ff2167722018c910e74e8 Oct 30 11:35:03.150742 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:03.150653 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" event={"ID":"c1a60678-72d4-453c-8c02-ea030952c1ce","Type":"ContainerStarted","Data":"4fd04b49bb8dfae30add8b4d4108a714d1c5910f04a6f7e19a3fe4110ee2c8c0"} Oct 30 11:35:03.150742 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:03.150709 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" event={"ID":"c1a60678-72d4-453c-8c02-ea030952c1ce","Type":"ContainerStarted","Data":"72842edaeb90b7e794b13e27f439fcf293c6a496091ff2167722018c910e74e8"} Oct 30 11:35:03.193028 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:03.192965 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" podStartSLOduration=3.192950658 podStartE2EDuration="3.192950658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:03.19232589 +0000 UTC m=+2993.693705376" watchObservedRunningTime="2026-10-30 11:35:03.192950658 +0000 UTC m=+2993.694330142" Oct 30 11:35:04.157895 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:04.157859 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" Oct 30 11:35:04.726898 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:04.726865 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:04.964493 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:04.964458 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:05.133335 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.133246 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:05.341966 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.341933 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 11:35:05.345735 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.345719 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:05.364252 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.364226 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 11:35:05.364392 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.364374 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 11:35:05.364392 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.364381 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 11:35:05.364495 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.364388 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 11:35:05.376119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.376032 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-kh4cb\"" Oct 30 11:35:05.378506 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.378479 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:35:05.383139 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.383116 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 11:35:05.383267 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.383250 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.395293 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.395256 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:05.403030 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.403008 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:35:05.431627 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.431596 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:05.431787 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.431630 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:05.431787 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.431652 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khwxs\" (UniqueName: \"kubernetes.io/projected/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-kube-api-access-khwxs\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:05.431787 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.431718 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:05.431910 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.431810 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:05.431910 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.431899 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:05.431987 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.431921 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:05.464693 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.464661 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 11:35:05.464693 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.464681 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 11:35:05.464893 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.464857 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 11:35:05.464893 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.464868 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-n5xr8\"" Oct 30 11:35:05.464893 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.464859 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 11:35:05.465022 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.464897 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 11:35:05.477558 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.477516 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 11:35:05.477558 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.477525 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 11:35:05.477736 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.477648 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 11:35:05.533085 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.533058 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:05.533237 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.533092 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk4dg\" (UniqueName: \"kubernetes.io/projected/b98eecff-d807-4b6c-a02e-d85763da9485-kube-api-access-wk4dg\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.533237 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.533114 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.533237 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.533140 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b98eecff-d807-4b6c-a02e-d85763da9485-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.533350 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.533271 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:05.533350 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.533303 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:05.533350 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.533333 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.533504 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.533363 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b98eecff-d807-4b6c-a02e-d85763da9485-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.533504 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.533458 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.533586 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.533528 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:05.533586 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.533571 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:05.533586 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.533578 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.533704 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.533605 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:05.533704 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.533634 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.533704 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.533666 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khwxs\" (UniqueName: \"kubernetes.io/projected/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-kube-api-access-khwxs\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:05.533704 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.533696 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:05.533901 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.533759 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b98eecff-d807-4b6c-a02e-d85763da9485-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.533901 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.533793 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b98eecff-d807-4b6c-a02e-d85763da9485-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.533901 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.533830 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.533901 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.533863 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b98eecff-d807-4b6c-a02e-d85763da9485-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.535974 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.535949 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:05.536085 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.536022 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:05.536085 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.536070 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:05.536267 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.536237 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:05.536376 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.536313 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:05.557374 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.557343 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khwxs\" (UniqueName: \"kubernetes.io/projected/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-kube-api-access-khwxs\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:05.561115 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.561088 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:05.634478 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.634424 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b98eecff-d807-4b6c-a02e-d85763da9485-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.634478 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.634477 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b98eecff-d807-4b6c-a02e-d85763da9485-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.634704 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.634510 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.634704 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.634534 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b98eecff-d807-4b6c-a02e-d85763da9485-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.634704 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.634559 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk4dg\" (UniqueName: \"kubernetes.io/projected/b98eecff-d807-4b6c-a02e-d85763da9485-kube-api-access-wk4dg\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.634704 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.634669 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.634911 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.634729 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b98eecff-d807-4b6c-a02e-d85763da9485-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.634911 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.634797 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.634911 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.634827 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b98eecff-d807-4b6c-a02e-d85763da9485-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.634911 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.634856 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.634911 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.634906 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.635187 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.634934 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.635187 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.635085 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b98eecff-d807-4b6c-a02e-d85763da9485-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.635187 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.635160 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b98eecff-d807-4b6c-a02e-d85763da9485-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.636142 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.636095 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b98eecff-d807-4b6c-a02e-d85763da9485-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.637615 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.637591 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b98eecff-d807-4b6c-a02e-d85763da9485-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.637859 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.637812 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b98eecff-d807-4b6c-a02e-d85763da9485-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.638196 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.638172 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.638279 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.638248 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.638323 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.638276 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.638323 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.638283 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.638455 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.638400 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.639263 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.639239 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.657863 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.657814 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk4dg\" (UniqueName: \"kubernetes.io/projected/b98eecff-d807-4b6c-a02e-d85763da9485-kube-api-access-wk4dg\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:05.663682 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.663662 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:05.980277 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:05.980237 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 11:35:05.982062 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:35:05.982034 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ec8e0d_9aa2_4a80_b352_b08b55a9d3e1.slice/crio-8494af37c6aefffd8caeb0400d448b1552cfed4fd26f650271ec2e12293d8337 WatchSource:0}: Error finding container 8494af37c6aefffd8caeb0400d448b1552cfed4fd26f650271ec2e12293d8337: Status 404 returned error can't find the container with id 8494af37c6aefffd8caeb0400d448b1552cfed4fd26f650271ec2e12293d8337 Oct 30 11:35:06.167168 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:06.167138 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1","Type":"ContainerStarted","Data":"8494af37c6aefffd8caeb0400d448b1552cfed4fd26f650271ec2e12293d8337"} Oct 30 11:35:07.173176 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:07.173136 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1","Type":"ContainerStarted","Data":"4199a66f2d10ef1c5f20c661c61199f9315ad17bc11d766a1760d8e0f13ca1e2"} Oct 30 11:35:07.654855 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:07.654813 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:07.684526 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:07.684499 4289 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:07.755643 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:07.755608 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:07.755821 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:07.755736 4289 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:07.770462 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:07.770414 4289 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") DevicePath \"csi-ea06c66241a0a003a5daff5d9fc602f40e53768b00b104c5e4118b92d558c090\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:07.871117 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:07.871075 4289 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7342da0537aa528cc1545d2018ac3fd0382dfe4f670067e58cf8ba34bb891174/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:07.897797 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:07.897770 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:08.105990 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:08.105958 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:08.436055 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:08.436026 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:35:08.437785 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:35:08.437759 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98eecff_d807_4b6c_a02e_d85763da9485.slice/crio-8e358cc3166cab4bc85e2411b8ed3a0620606b483ead647e4db0fa85e166a045 WatchSource:0}: Error finding container 8e358cc3166cab4bc85e2411b8ed3a0620606b483ead647e4db0fa85e166a045: Status 404 returned error can't find the container with id 8e358cc3166cab4bc85e2411b8ed3a0620606b483ead647e4db0fa85e166a045 Oct 30 11:35:09.192624 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:09.192536 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b98eecff-d807-4b6c-a02e-d85763da9485","Type":"ContainerStarted","Data":"7b89ddd71cfe1f630782a4de1d6037f59c5f52f8ac53800125eaeb7ce7244cbd"} Oct 30 11:35:09.192624 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:09.192596 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b98eecff-d807-4b6c-a02e-d85763da9485","Type":"ContainerStarted","Data":"8e358cc3166cab4bc85e2411b8ed3a0620606b483ead647e4db0fa85e166a045"} Oct 30 11:35:10.164528 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:10.164448 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" Oct 30 11:35:10.198882 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:10.198849 4289 generic.go:356] "Generic (PLEG): container finished" podID="07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1" containerID="4199a66f2d10ef1c5f20c661c61199f9315ad17bc11d766a1760d8e0f13ca1e2" exitCode=0 Oct 30 11:35:10.199045 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:10.198921 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1","Type":"ContainerDied","Data":"4199a66f2d10ef1c5f20c661c61199f9315ad17bc11d766a1760d8e0f13ca1e2"} Oct 30 11:35:11.212630 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:11.212553 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1","Type":"ContainerStarted","Data":"83bddfda9583cdc7934fb3ad3855da07edffc3849260129d131867e78ae7d9a7"} Oct 30 11:35:11.212993 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:11.212638 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1","Type":"ContainerStarted","Data":"1e534681b23cba377c3323dcbfb8390e73db95b8be96b633216ba856fbead16a"} Oct 30 11:35:11.212993 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:11.212848 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:11.260681 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:11.260627 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=2.8757291990000002 podStartE2EDuration="6.260614472s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.384885273s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:11.259725225 +0000 UTC m=+3001.761104710" watchObservedRunningTime="2026-10-30 11:35:11.260614472 +0000 UTC m=+3001.761993957" Oct 30 11:35:13.154003 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:13.153972 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz" Oct 30 11:35:13.222941 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:13.222913 4289 generic.go:356] "Generic (PLEG): container finished" podID="b98eecff-d807-4b6c-a02e-d85763da9485" containerID="7b89ddd71cfe1f630782a4de1d6037f59c5f52f8ac53800125eaeb7ce7244cbd" exitCode=0 Oct 30 11:35:13.223103 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:13.222976 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b98eecff-d807-4b6c-a02e-d85763da9485","Type":"ContainerDied","Data":"7b89ddd71cfe1f630782a4de1d6037f59c5f52f8ac53800125eaeb7ce7244cbd"} Oct 30 11:35:14.229779 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:14.229744 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b98eecff-d807-4b6c-a02e-d85763da9485","Type":"ContainerStarted","Data":"958b8b883b30e675c4d8cb9795253c467703e49b36bec654f2172a7c17e40ac7"} Oct 30 11:35:14.230197 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:14.229814 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b98eecff-d807-4b6c-a02e-d85763da9485","Type":"ContainerStarted","Data":"a7e015a68cf0961b3b10edaeaf3735346099c35b12725573dca1e9637014c666"} Oct 30 11:35:14.230197 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:14.229846 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b98eecff-d807-4b6c-a02e-d85763da9485","Type":"ContainerStarted","Data":"229cec0536cb47ce8f03f713a5977fc603a9aea9bd04a81c3125a8eb19831d2d"} Oct 30 11:35:14.320234 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:14.320184 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=5.6359007519999995 podStartE2EDuration="9.320171468s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.684270716s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:14.306359867 +0000 UTC m=+3004.807739352" watchObservedRunningTime="2026-10-30 11:35:14.320171468 +0000 UTC m=+3004.821550953" Oct 30 11:35:17.217943 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:17.217914 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:18.106473 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:18.106420 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:19.841339 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:19.841298 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:20.040881 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:20.040836 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:21.552835 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:21.552785 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:21.719581 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:21.719524 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:21.888192 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:21.888111 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:22.055751 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.055700 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:22.155918 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.155839 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:35:22.159694 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.159676 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:35:22.188923 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.188895 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:35:22.189300 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.189277 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvctk\" (UniqueName: \"kubernetes.io/projected/85f4e48a-f1de-4fe7-a113-c096686d57bb-kube-api-access-fvctk\") pod \"data-science-collector-collector-1\" (UID: \"85f4e48a-f1de-4fe7-a113-c096686d57bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:35:22.189356 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.189319 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/85f4e48a-f1de-4fe7-a113-c096686d57bb-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"85f4e48a-f1de-4fe7-a113-c096686d57bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:35:22.189411 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.189397 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85f4e48a-f1de-4fe7-a113-c096686d57bb-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"85f4e48a-f1de-4fe7-a113-c096686d57bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:35:22.189483 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.189452 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/85f4e48a-f1de-4fe7-a113-c096686d57bb-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"85f4e48a-f1de-4fe7-a113-c096686d57bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:35:22.189586 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.189565 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/85f4e48a-f1de-4fe7-a113-c096686d57bb-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"85f4e48a-f1de-4fe7-a113-c096686d57bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:35:22.193825 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.193804 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-e4cadd38\"" Oct 30 11:35:22.218492 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.218467 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 11:35:22.218693 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.218667 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 11:35:22.218765 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.218690 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 11:35:22.218765 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.218707 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-7p6tb\"" Oct 30 11:35:22.290932 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.290896 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/85f4e48a-f1de-4fe7-a113-c096686d57bb-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"85f4e48a-f1de-4fe7-a113-c096686d57bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:35:22.291129 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.290946 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85f4e48a-f1de-4fe7-a113-c096686d57bb-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"85f4e48a-f1de-4fe7-a113-c096686d57bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:35:22.291129 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.290981 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/85f4e48a-f1de-4fe7-a113-c096686d57bb-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"85f4e48a-f1de-4fe7-a113-c096686d57bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:35:22.291129 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.291030 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/85f4e48a-f1de-4fe7-a113-c096686d57bb-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"85f4e48a-f1de-4fe7-a113-c096686d57bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:35:22.291129 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.291101 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvctk\" (UniqueName: \"kubernetes.io/projected/85f4e48a-f1de-4fe7-a113-c096686d57bb-kube-api-access-fvctk\") pod \"data-science-collector-collector-1\" (UID: \"85f4e48a-f1de-4fe7-a113-c096686d57bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:35:22.292062 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.292033 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/85f4e48a-f1de-4fe7-a113-c096686d57bb-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"85f4e48a-f1de-4fe7-a113-c096686d57bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:35:22.293778 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.293751 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/85f4e48a-f1de-4fe7-a113-c096686d57bb-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"85f4e48a-f1de-4fe7-a113-c096686d57bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:35:22.293939 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.293918 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/85f4e48a-f1de-4fe7-a113-c096686d57bb-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"85f4e48a-f1de-4fe7-a113-c096686d57bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:35:22.294005 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.293934 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85f4e48a-f1de-4fe7-a113-c096686d57bb-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"85f4e48a-f1de-4fe7-a113-c096686d57bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:35:22.314492 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.314462 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvctk\" (UniqueName: \"kubernetes.io/projected/85f4e48a-f1de-4fe7-a113-c096686d57bb-kube-api-access-fvctk\") pod \"data-science-collector-collector-1\" (UID: \"85f4e48a-f1de-4fe7-a113-c096686d57bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:35:22.472498 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.472469 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:35:22.775407 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.775378 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:35:22.776882 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:35:22.776853 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f4e48a_f1de_4fe7_a113_c096686d57bb.slice/crio-396a988ca5b15c2b3a287bbfabf1faf80b63c4916f0e51c3fbc2c64920918b0a WatchSource:0}: Error finding container 396a988ca5b15c2b3a287bbfabf1faf80b63c4916f0e51c3fbc2c64920918b0a: Status 404 returned error can't find the container with id 396a988ca5b15c2b3a287bbfabf1faf80b63c4916f0e51c3fbc2c64920918b0a Oct 30 11:35:22.778983 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.778948 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:35:22.779087 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:22.779021 4289 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 11:35:23.107246 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:23.107160 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:23.112852 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:23.112825 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:23.271537 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:23.271499 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"85f4e48a-f1de-4fe7-a113-c096686d57bb","Type":"ContainerStarted","Data":"5e7c0caa438d7abb9c892090ca2177c694d106289ff199c4a073a807e74a6831"} Oct 30 11:35:23.271726 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:23.271566 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"85f4e48a-f1de-4fe7-a113-c096686d57bb","Type":"ContainerStarted","Data":"396a988ca5b15c2b3a287bbfabf1faf80b63c4916f0e51c3fbc2c64920918b0a"} Oct 30 11:35:23.276097 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:23.276075 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:23.310086 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:23.310037 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.310023799 podStartE2EDuration="1.310023799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:23.309193171 +0000 UTC m=+3013.810572659" watchObservedRunningTime="2026-10-30 11:35:23.310023799 +0000 UTC m=+3013.811403284" Oct 30 11:35:26.257930 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:35:26.257891 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:35:30.521090 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:30.521056 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:30.721129 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:30.721097 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:30.884623 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:30.884527 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:32.145214 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:32.145176 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:35:32.145682 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:32.145421 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="85f4e48a-f1de-4fe7-a113-c096686d57bb" containerName="otc-container" containerID="cri-o://5e7c0caa438d7abb9c892090ca2177c694d106289ff199c4a073a807e74a6831" gracePeriod=30 Oct 30 11:35:32.323192 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:32.323147 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:32.491909 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:32.491868 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:33.325745 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:33.325713 4289 generic.go:356] "Generic (PLEG): container finished" podID="85f4e48a-f1de-4fe7-a113-c096686d57bb" containerID="5e7c0caa438d7abb9c892090ca2177c694d106289ff199c4a073a807e74a6831" exitCode=0 Oct 30 11:35:33.326094 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:33.325783 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"85f4e48a-f1de-4fe7-a113-c096686d57bb","Type":"ContainerDied","Data":"5e7c0caa438d7abb9c892090ca2177c694d106289ff199c4a073a807e74a6831"} Oct 30 11:35:33.390116 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:33.390087 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:35:33.497732 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:33.497643 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f4e48a-f1de-4fe7-a113-c096686d57bb-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/85f4e48a-f1de-4fe7-a113-c096686d57bb-monitoring-tls-certs\") pod \"85f4e48a-f1de-4fe7-a113-c096686d57bb\" (UID: \"85f4e48a-f1de-4fe7-a113-c096686d57bb\") " Oct 30 11:35:33.497732 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:33.497694 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f4e48a-f1de-4fe7-a113-c096686d57bb-kube-api-access-fvctk\" (UniqueName: \"kubernetes.io/projected/85f4e48a-f1de-4fe7-a113-c096686d57bb-kube-api-access-fvctk\") pod \"85f4e48a-f1de-4fe7-a113-c096686d57bb\" (UID: \"85f4e48a-f1de-4fe7-a113-c096686d57bb\") " Oct 30 11:35:33.497926 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:33.497741 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f4e48a-f1de-4fe7-a113-c096686d57bb-otc-internal\" (UniqueName: \"kubernetes.io/configmap/85f4e48a-f1de-4fe7-a113-c096686d57bb-otc-internal\") pod \"85f4e48a-f1de-4fe7-a113-c096686d57bb\" (UID: \"85f4e48a-f1de-4fe7-a113-c096686d57bb\") " Oct 30 11:35:33.497926 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:33.497788 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f4e48a-f1de-4fe7-a113-c096686d57bb-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/85f4e48a-f1de-4fe7-a113-c096686d57bb-data-science-collector-ta-client-cert\") pod \"85f4e48a-f1de-4fe7-a113-c096686d57bb\" (UID: \"85f4e48a-f1de-4fe7-a113-c096686d57bb\") " Oct 30 11:35:33.497926 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:33.497843 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f4e48a-f1de-4fe7-a113-c096686d57bb-tls-certs\" (UniqueName: \"kubernetes.io/secret/85f4e48a-f1de-4fe7-a113-c096686d57bb-tls-certs\") pod \"85f4e48a-f1de-4fe7-a113-c096686d57bb\" (UID: \"85f4e48a-f1de-4fe7-a113-c096686d57bb\") " Oct 30 11:35:33.498139 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:33.498109 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f4e48a-f1de-4fe7-a113-c096686d57bb-otc-internal" pod "85f4e48a-f1de-4fe7-a113-c096686d57bb" (UID: "85f4e48a-f1de-4fe7-a113-c096686d57bb"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:35:33.500115 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:33.500081 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f4e48a-f1de-4fe7-a113-c096686d57bb-monitoring-tls-certs" pod "85f4e48a-f1de-4fe7-a113-c096686d57bb" (UID: "85f4e48a-f1de-4fe7-a113-c096686d57bb"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:35:33.500115 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:33.500088 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f4e48a-f1de-4fe7-a113-c096686d57bb-kube-api-access-fvctk" pod "85f4e48a-f1de-4fe7-a113-c096686d57bb" (UID: "85f4e48a-f1de-4fe7-a113-c096686d57bb"). InnerVolumeSpecName "kube-api-access-fvctk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:35:33.500319 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:33.500118 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f4e48a-f1de-4fe7-a113-c096686d57bb-data-science-collector-ta-client-cert" pod "85f4e48a-f1de-4fe7-a113-c096686d57bb" (UID: "85f4e48a-f1de-4fe7-a113-c096686d57bb"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:35:33.500319 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:33.500130 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f4e48a-f1de-4fe7-a113-c096686d57bb-tls-certs" pod "85f4e48a-f1de-4fe7-a113-c096686d57bb" (UID: "85f4e48a-f1de-4fe7-a113-c096686d57bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:35:33.599289 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:33.599253 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvctk\" (UniqueName: \"kubernetes.io/projected/85f4e48a-f1de-4fe7-a113-c096686d57bb-kube-api-access-fvctk\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:33.599289 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:33.599282 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/85f4e48a-f1de-4fe7-a113-c096686d57bb-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:33.599289 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:33.599293 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/85f4e48a-f1de-4fe7-a113-c096686d57bb-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:33.599559 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:33.599302 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85f4e48a-f1de-4fe7-a113-c096686d57bb-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:33.599559 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:33.599310 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/85f4e48a-f1de-4fe7-a113-c096686d57bb-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:34.331958 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:34.331929 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 11:35:34.332404 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:34.331923 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"85f4e48a-f1de-4fe7-a113-c096686d57bb","Type":"ContainerDied","Data":"396a988ca5b15c2b3a287bbfabf1faf80b63c4916f0e51c3fbc2c64920918b0a"} Oct 30 11:35:34.332404 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:34.332072 4289 scope.go:122] "RemoveContainer" containerID="5e7c0caa438d7abb9c892090ca2177c694d106289ff199c4a073a807e74a6831" Oct 30 11:35:34.381808 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:34.381780 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:35:34.393788 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:34.393758 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 11:35:35.625295 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:35.625260 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f4e48a-f1de-4fe7-a113-c096686d57bb" path="/var/lib/kubelet/pods/85f4e48a-f1de-4fe7-a113-c096686d57bb/volumes" Oct 30 11:35:36.052592 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.052550 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6"] Oct 30 11:35:36.052819 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.052795 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" podUID="c1a60678-72d4-453c-8c02-ea030952c1ce" containerName="kube-rbac-proxy" containerID="cri-o://4fd04b49bb8dfae30add8b4d4108a714d1c5910f04a6f7e19a3fe4110ee2c8c0" gracePeriod=30 Oct 30 11:35:36.064623 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.064596 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd"] Oct 30 11:35:36.064880 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.064855 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" podUID="c23b2839-9e2a-4465-8825-ab1a1d82dada" containerName="kube-rbac-proxy" containerID="cri-o://19e0f03fa4e00667e1cca4bf102c8abb1ca96f9b0e5b1677f8adac76e4954ec6" gracePeriod=30 Oct 30 11:35:36.064977 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.064874 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" podUID="c23b2839-9e2a-4465-8825-ab1a1d82dada" containerName="prom-label-proxy" containerID="cri-o://a3dd94f4c2a4581caf4c608afc5aad0fe0f7c8c5b9fc0680a59572c7f18aec2c" gracePeriod=30 Oct 30 11:35:36.065038 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.064949 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" podUID="c23b2839-9e2a-4465-8825-ab1a1d82dada" containerName="method-gate" containerID="cri-o://00276833e29815457b8fdea7aaa0d696f658648f35ffee255d8f7e9becbf8906" gracePeriod=30 Oct 30 11:35:36.081543 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.081518 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz"] Oct 30 11:35:36.081741 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.081724 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" podUID="708f844f-1c03-4e6f-95b9-7f3b8dabff06" containerName="kube-rbac-proxy" containerID="cri-o://ec4ea8bc43eef93b09798857da7657d14e597d594cfcfcff6f7d3b515033bd43" gracePeriod=30 Oct 30 11:35:36.081829 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.081803 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" podUID="708f844f-1c03-4e6f-95b9-7f3b8dabff06" containerName="prom-label-proxy" containerID="cri-o://bf7e5ead5ca9a3126391e7bba5b706e9c922675b9aec21a900e9a0633c26ba64" gracePeriod=30 Oct 30 11:35:36.081917 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.081810 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" podUID="708f844f-1c03-4e6f-95b9-7f3b8dabff06" containerName="method-gate" containerID="cri-o://24698e8db6f2a06f965ed6cbcc4dfd67ef3c0ad979c3a0b304d243665277957e" gracePeriod=30 Oct 30 11:35:36.092210 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:35:36.092184 4289 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a60678_72d4_453c_8c02_ea030952c1ce.slice/crio-4fd04b49bb8dfae30add8b4d4108a714d1c5910f04a6f7e19a3fe4110ee2c8c0.scope\": RecentStats: unable to find data in memory cache]" Oct 30 11:35:36.095973 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.095952 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz"] Oct 30 11:35:36.096136 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.096118 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz" podUID="ff9592ba-475a-4ced-b3a2-92694daf0e15" containerName="korrel8r" containerID="cri-o://428f96ce2cee38489f5882fb90d1a118956a227929f67ee5291ee9c3e3bb1805" gracePeriod=30 Oct 30 11:35:36.345389 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.345357 4289 generic.go:356] "Generic (PLEG): container finished" podID="c23b2839-9e2a-4465-8825-ab1a1d82dada" containerID="a3dd94f4c2a4581caf4c608afc5aad0fe0f7c8c5b9fc0680a59572c7f18aec2c" exitCode=0 Oct 30 11:35:36.345389 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.345385 4289 generic.go:356] "Generic (PLEG): container finished" podID="c23b2839-9e2a-4465-8825-ab1a1d82dada" containerID="00276833e29815457b8fdea7aaa0d696f658648f35ffee255d8f7e9becbf8906" exitCode=0 Oct 30 11:35:36.345389 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.345392 4289 generic.go:356] "Generic (PLEG): container finished" podID="c23b2839-9e2a-4465-8825-ab1a1d82dada" containerID="19e0f03fa4e00667e1cca4bf102c8abb1ca96f9b0e5b1677f8adac76e4954ec6" exitCode=0 Oct 30 11:35:36.345632 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.345464 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" event={"ID":"c23b2839-9e2a-4465-8825-ab1a1d82dada","Type":"ContainerDied","Data":"a3dd94f4c2a4581caf4c608afc5aad0fe0f7c8c5b9fc0680a59572c7f18aec2c"} Oct 30 11:35:36.345632 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.345589 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" event={"ID":"c23b2839-9e2a-4465-8825-ab1a1d82dada","Type":"ContainerDied","Data":"00276833e29815457b8fdea7aaa0d696f658648f35ffee255d8f7e9becbf8906"} Oct 30 11:35:36.345700 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.345633 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" event={"ID":"c23b2839-9e2a-4465-8825-ab1a1d82dada","Type":"ContainerDied","Data":"19e0f03fa4e00667e1cca4bf102c8abb1ca96f9b0e5b1677f8adac76e4954ec6"} Oct 30 11:35:36.347721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.347697 4289 generic.go:356] "Generic (PLEG): container finished" podID="708f844f-1c03-4e6f-95b9-7f3b8dabff06" containerID="bf7e5ead5ca9a3126391e7bba5b706e9c922675b9aec21a900e9a0633c26ba64" exitCode=0 Oct 30 11:35:36.347721 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.347718 4289 generic.go:356] "Generic (PLEG): container finished" podID="708f844f-1c03-4e6f-95b9-7f3b8dabff06" containerID="24698e8db6f2a06f965ed6cbcc4dfd67ef3c0ad979c3a0b304d243665277957e" exitCode=0 Oct 30 11:35:36.347879 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.347730 4289 generic.go:356] "Generic (PLEG): container finished" podID="708f844f-1c03-4e6f-95b9-7f3b8dabff06" containerID="ec4ea8bc43eef93b09798857da7657d14e597d594cfcfcff6f7d3b515033bd43" exitCode=0 Oct 30 11:35:36.347879 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.347772 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" event={"ID":"708f844f-1c03-4e6f-95b9-7f3b8dabff06","Type":"ContainerDied","Data":"bf7e5ead5ca9a3126391e7bba5b706e9c922675b9aec21a900e9a0633c26ba64"} Oct 30 11:35:36.347879 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.347824 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" event={"ID":"708f844f-1c03-4e6f-95b9-7f3b8dabff06","Type":"ContainerDied","Data":"24698e8db6f2a06f965ed6cbcc4dfd67ef3c0ad979c3a0b304d243665277957e"} Oct 30 11:35:36.347879 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.347862 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" event={"ID":"708f844f-1c03-4e6f-95b9-7f3b8dabff06","Type":"ContainerDied","Data":"ec4ea8bc43eef93b09798857da7657d14e597d594cfcfcff6f7d3b515033bd43"} Oct 30 11:35:36.349890 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.349871 4289 generic.go:356] "Generic (PLEG): container finished" podID="ff9592ba-475a-4ced-b3a2-92694daf0e15" containerID="428f96ce2cee38489f5882fb90d1a118956a227929f67ee5291ee9c3e3bb1805" exitCode=0 Oct 30 11:35:36.349976 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.349954 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz" event={"ID":"ff9592ba-475a-4ced-b3a2-92694daf0e15","Type":"ContainerDied","Data":"428f96ce2cee38489f5882fb90d1a118956a227929f67ee5291ee9c3e3bb1805"} Oct 30 11:35:36.351338 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.351317 4289 generic.go:356] "Generic (PLEG): container finished" podID="c1a60678-72d4-453c-8c02-ea030952c1ce" containerID="4fd04b49bb8dfae30add8b4d4108a714d1c5910f04a6f7e19a3fe4110ee2c8c0" exitCode=0 Oct 30 11:35:36.351422 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.351361 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" event={"ID":"c1a60678-72d4-453c-8c02-ea030952c1ce","Type":"ContainerDied","Data":"4fd04b49bb8dfae30add8b4d4108a714d1c5910f04a6f7e19a3fe4110ee2c8c0"} Oct 30 11:35:36.358988 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.357815 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" Oct 30 11:35:36.437948 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.437132 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" Oct 30 11:35:36.437948 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.437572 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-tls\" (UniqueName: \"kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-tls\") pod \"c1a60678-72d4-453c-8c02-ea030952c1ce\" (UID: \"c1a60678-72d4-453c-8c02-ea030952c1ce\") " Oct 30 11:35:36.437948 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.437663 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1a60678-72d4-453c-8c02-ea030952c1ce-kube-api-access-kcwrt\" (UniqueName: \"kubernetes.io/projected/c1a60678-72d4-453c-8c02-ea030952c1ce-kube-api-access-kcwrt\") pod \"c1a60678-72d4-453c-8c02-ea030952c1ce\" (UID: \"c1a60678-72d4-453c-8c02-ea030952c1ce\") " Oct 30 11:35:36.437948 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.437743 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-prometheus-client-cert\") pod \"c1a60678-72d4-453c-8c02-ea030952c1ce\" (UID: \"c1a60678-72d4-453c-8c02-ea030952c1ce\") " Oct 30 11:35:36.437948 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.437789 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1a60678-72d4-453c-8c02-ea030952c1ce-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/c1a60678-72d4-453c-8c02-ea030952c1ce-prometheus-ca\") pod \"c1a60678-72d4-453c-8c02-ea030952c1ce\" (UID: \"c1a60678-72d4-453c-8c02-ea030952c1ce\") " Oct 30 11:35:36.437948 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.437832 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-kube-rbac-proxy-config\") pod \"c1a60678-72d4-453c-8c02-ea030952c1ce\" (UID: \"c1a60678-72d4-453c-8c02-ea030952c1ce\") " Oct 30 11:35:36.440107 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.440053 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a60678-72d4-453c-8c02-ea030952c1ce-prometheus-ca" pod "c1a60678-72d4-453c-8c02-ea030952c1ce" (UID: "c1a60678-72d4-453c-8c02-ea030952c1ce"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:35:36.442339 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.442307 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a60678-72d4-453c-8c02-ea030952c1ce-kube-api-access-kcwrt" pod "c1a60678-72d4-453c-8c02-ea030952c1ce" (UID: "c1a60678-72d4-453c-8c02-ea030952c1ce"). InnerVolumeSpecName "kube-api-access-kcwrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:35:36.443812 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.443563 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-prometheus-client-cert" pod "c1a60678-72d4-453c-8c02-ea030952c1ce" (UID: "c1a60678-72d4-453c-8c02-ea030952c1ce"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:35:36.443812 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.443588 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-tls" pod "c1a60678-72d4-453c-8c02-ea030952c1ce" (UID: "c1a60678-72d4-453c-8c02-ea030952c1ce"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:35:36.443812 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.443603 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-kube-rbac-proxy-config" pod "c1a60678-72d4-453c-8c02-ea030952c1ce" (UID: "c1a60678-72d4-453c-8c02-ea030952c1ce"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:35:36.446038 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.445996 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" Oct 30 11:35:36.453673 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.453649 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz" Oct 30 11:35:36.495165 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.495134 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:36.539066 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.539033 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c23b2839-9e2a-4465-8825-ab1a1d82dada-config\" (UniqueName: \"kubernetes.io/configmap/c23b2839-9e2a-4465-8825-ab1a1d82dada-config\") pod \"c23b2839-9e2a-4465-8825-ab1a1d82dada\" (UID: \"c23b2839-9e2a-4465-8825-ab1a1d82dada\") " Oct 30 11:35:36.539264 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.539081 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c23b2839-9e2a-4465-8825-ab1a1d82dada-client-ca\" (UniqueName: \"kubernetes.io/configmap/c23b2839-9e2a-4465-8825-ab1a1d82dada-client-ca\") pod \"c23b2839-9e2a-4465-8825-ab1a1d82dada\" (UID: \"c23b2839-9e2a-4465-8825-ab1a1d82dada\") " Oct 30 11:35:36.539264 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.539120 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708f844f-1c03-4e6f-95b9-7f3b8dabff06-kube-api-access-sz89t\" (UniqueName: \"kubernetes.io/projected/708f844f-1c03-4e6f-95b9-7f3b8dabff06-kube-api-access-sz89t\") pod \"708f844f-1c03-4e6f-95b9-7f3b8dabff06\" (UID: \"708f844f-1c03-4e6f-95b9-7f3b8dabff06\") " Oct 30 11:35:36.539264 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.539159 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c23b2839-9e2a-4465-8825-ab1a1d82dada-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/c23b2839-9e2a-4465-8825-ab1a1d82dada-prometheus-tls-ca\") pod \"c23b2839-9e2a-4465-8825-ab1a1d82dada\" (UID: \"c23b2839-9e2a-4465-8825-ab1a1d82dada\") " Oct 30 11:35:36.539264 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.539190 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/708f844f-1c03-4e6f-95b9-7f3b8dabff06-config\" (UniqueName: \"kubernetes.io/configmap/708f844f-1c03-4e6f-95b9-7f3b8dabff06-config\") pod \"708f844f-1c03-4e6f-95b9-7f3b8dabff06\" (UID: \"708f844f-1c03-4e6f-95b9-7f3b8dabff06\") " Oct 30 11:35:36.539264 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.539227 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff9592ba-475a-4ced-b3a2-92694daf0e15-tls\" (UniqueName: \"kubernetes.io/secret/ff9592ba-475a-4ced-b3a2-92694daf0e15-tls\") pod \"ff9592ba-475a-4ced-b3a2-92694daf0e15\" (UID: \"ff9592ba-475a-4ced-b3a2-92694daf0e15\") " Oct 30 11:35:36.539264 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.539253 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff9592ba-475a-4ced-b3a2-92694daf0e15-kube-api-access-nqms5\" (UniqueName: \"kubernetes.io/projected/ff9592ba-475a-4ced-b3a2-92694daf0e15-kube-api-access-nqms5\") pod \"ff9592ba-475a-4ced-b3a2-92694daf0e15\" (UID: \"ff9592ba-475a-4ced-b3a2-92694daf0e15\") " Oct 30 11:35:36.539649 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.539280 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff9592ba-475a-4ced-b3a2-92694daf0e15-config\" (UniqueName: \"kubernetes.io/configmap/ff9592ba-475a-4ced-b3a2-92694daf0e15-config\") pod \"ff9592ba-475a-4ced-b3a2-92694daf0e15\" (UID: \"ff9592ba-475a-4ced-b3a2-92694daf0e15\") " Oct 30 11:35:36.539649 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.539317 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/708f844f-1c03-4e6f-95b9-7f3b8dabff06-tls\" (UniqueName: \"kubernetes.io/secret/708f844f-1c03-4e6f-95b9-7f3b8dabff06-tls\") pod \"708f844f-1c03-4e6f-95b9-7f3b8dabff06\" (UID: \"708f844f-1c03-4e6f-95b9-7f3b8dabff06\") " Oct 30 11:35:36.539649 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.539353 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c23b2839-9e2a-4465-8825-ab1a1d82dada-kube-api-access-2pq24\" (UniqueName: \"kubernetes.io/projected/c23b2839-9e2a-4465-8825-ab1a1d82dada-kube-api-access-2pq24\") pod \"c23b2839-9e2a-4465-8825-ab1a1d82dada\" (UID: \"c23b2839-9e2a-4465-8825-ab1a1d82dada\") " Oct 30 11:35:36.539649 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.539462 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c23b2839-9e2a-4465-8825-ab1a1d82dada-tls\" (UniqueName: \"kubernetes.io/secret/c23b2839-9e2a-4465-8825-ab1a1d82dada-tls\") pod \"c23b2839-9e2a-4465-8825-ab1a1d82dada\" (UID: \"c23b2839-9e2a-4465-8825-ab1a1d82dada\") " Oct 30 11:35:36.539870 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.539759 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-prometheus-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:36.539870 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.539777 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/c1a60678-72d4-453c-8c02-ea030952c1ce-prometheus-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:36.539870 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.539794 4289 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-kube-rbac-proxy-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:36.539870 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.539808 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c1a60678-72d4-453c-8c02-ea030952c1ce-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:36.539870 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.539823 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcwrt\" (UniqueName: \"kubernetes.io/projected/c1a60678-72d4-453c-8c02-ea030952c1ce-kube-api-access-kcwrt\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:36.540137 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.540036 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708f844f-1c03-4e6f-95b9-7f3b8dabff06-config" pod "708f844f-1c03-4e6f-95b9-7f3b8dabff06" (UID: "708f844f-1c03-4e6f-95b9-7f3b8dabff06"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:35:36.540248 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.540223 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff9592ba-475a-4ced-b3a2-92694daf0e15-config" pod "ff9592ba-475a-4ced-b3a2-92694daf0e15" (UID: "ff9592ba-475a-4ced-b3a2-92694daf0e15"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:35:36.540919 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.540752 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c23b2839-9e2a-4465-8825-ab1a1d82dada-client-ca" pod "c23b2839-9e2a-4465-8825-ab1a1d82dada" (UID: "c23b2839-9e2a-4465-8825-ab1a1d82dada"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:35:36.540919 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.540879 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c23b2839-9e2a-4465-8825-ab1a1d82dada-prometheus-tls-ca" pod "c23b2839-9e2a-4465-8825-ab1a1d82dada" (UID: "c23b2839-9e2a-4465-8825-ab1a1d82dada"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:35:36.541093 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.541064 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c23b2839-9e2a-4465-8825-ab1a1d82dada-config" pod "c23b2839-9e2a-4465-8825-ab1a1d82dada" (UID: "c23b2839-9e2a-4465-8825-ab1a1d82dada"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:35:36.542537 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.542505 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708f844f-1c03-4e6f-95b9-7f3b8dabff06-tls" pod "708f844f-1c03-4e6f-95b9-7f3b8dabff06" (UID: "708f844f-1c03-4e6f-95b9-7f3b8dabff06"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:35:36.542704 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.542673 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff9592ba-475a-4ced-b3a2-92694daf0e15-kube-api-access-nqms5" pod "ff9592ba-475a-4ced-b3a2-92694daf0e15" (UID: "ff9592ba-475a-4ced-b3a2-92694daf0e15"). InnerVolumeSpecName "kube-api-access-nqms5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:35:36.542904 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.542880 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708f844f-1c03-4e6f-95b9-7f3b8dabff06-kube-api-access-sz89t" pod "708f844f-1c03-4e6f-95b9-7f3b8dabff06" (UID: "708f844f-1c03-4e6f-95b9-7f3b8dabff06"). InnerVolumeSpecName "kube-api-access-sz89t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:35:36.543138 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.543119 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c23b2839-9e2a-4465-8825-ab1a1d82dada-kube-api-access-2pq24" pod "c23b2839-9e2a-4465-8825-ab1a1d82dada" (UID: "c23b2839-9e2a-4465-8825-ab1a1d82dada"). InnerVolumeSpecName "kube-api-access-2pq24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:35:36.544004 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.543956 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9592ba-475a-4ced-b3a2-92694daf0e15-tls" pod "ff9592ba-475a-4ced-b3a2-92694daf0e15" (UID: "ff9592ba-475a-4ced-b3a2-92694daf0e15"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:35:36.546276 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.546219 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c23b2839-9e2a-4465-8825-ab1a1d82dada-tls" pod "c23b2839-9e2a-4465-8825-ab1a1d82dada" (UID: "c23b2839-9e2a-4465-8825-ab1a1d82dada"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:35:36.575386 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.575351 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 11:35:36.575614 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.575588 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1" containerName="alertmanager" containerID="cri-o://1e534681b23cba377c3323dcbfb8390e73db95b8be96b633216ba856fbead16a" gracePeriod=120 Oct 30 11:35:36.575920 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.575898 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1" containerName="config-reloader" containerID="cri-o://83bddfda9583cdc7934fb3ad3855da07edffc3849260129d131867e78ae7d9a7" gracePeriod=120 Oct 30 11:35:36.588462 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.588371 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:35:36.588754 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.588723 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="b98eecff-d807-4b6c-a02e-d85763da9485" containerName="prometheus" containerID="cri-o://229cec0536cb47ce8f03f713a5977fc603a9aea9bd04a81c3125a8eb19831d2d" gracePeriod=600 Oct 30 11:35:36.588754 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.588744 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="b98eecff-d807-4b6c-a02e-d85763da9485" containerName="config-reloader" containerID="cri-o://a7e015a68cf0961b3b10edaeaf3735346099c35b12725573dca1e9637014c666" gracePeriod=600 Oct 30 11:35:36.588991 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.588744 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="b98eecff-d807-4b6c-a02e-d85763da9485" containerName="thanos-sidecar" containerID="cri-o://958b8b883b30e675c4d8cb9795253c467703e49b36bec654f2172a7c17e40ac7" gracePeriod=600 Oct 30 11:35:36.640951 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.640923 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c23b2839-9e2a-4465-8825-ab1a1d82dada-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:36.640951 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.640947 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c23b2839-9e2a-4465-8825-ab1a1d82dada-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:36.640951 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.640955 4289 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c23b2839-9e2a-4465-8825-ab1a1d82dada-client-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:36.641354 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.640965 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz89t\" (UniqueName: \"kubernetes.io/projected/708f844f-1c03-4e6f-95b9-7f3b8dabff06-kube-api-access-sz89t\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:36.641354 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.640974 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/c23b2839-9e2a-4465-8825-ab1a1d82dada-prometheus-tls-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:36.641354 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.640982 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/708f844f-1c03-4e6f-95b9-7f3b8dabff06-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:36.641354 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.640989 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ff9592ba-475a-4ced-b3a2-92694daf0e15-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:36.641354 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.640997 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqms5\" (UniqueName: \"kubernetes.io/projected/ff9592ba-475a-4ced-b3a2-92694daf0e15-kube-api-access-nqms5\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:36.641354 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.641006 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff9592ba-475a-4ced-b3a2-92694daf0e15-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:36.641354 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.641014 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/708f844f-1c03-4e6f-95b9-7f3b8dabff06-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:36.641354 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.641022 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pq24\" (UniqueName: \"kubernetes.io/projected/c23b2839-9e2a-4465-8825-ab1a1d82dada-kube-api-access-2pq24\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:36.681039 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.681005 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:36.854888 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.854806 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:36.951808 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.951773 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 11:35:36.952143 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952130 4289 cpu_manager.go:414] "containerMap: removing container" podUID="ff9592ba-475a-4ced-b3a2-92694daf0e15" containerName="korrel8r" Oct 30 11:35:36.952187 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952144 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff9592ba-475a-4ced-b3a2-92694daf0e15" containerName="korrel8r" Oct 30 11:35:36.952187 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952157 4289 cpu_manager.go:414] "containerMap: removing container" podUID="c1a60678-72d4-453c-8c02-ea030952c1ce" containerName="kube-rbac-proxy" Oct 30 11:35:36.952187 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952163 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1a60678-72d4-453c-8c02-ea030952c1ce" containerName="kube-rbac-proxy" Oct 30 11:35:36.952187 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952172 4289 cpu_manager.go:414] "containerMap: removing container" podUID="c23b2839-9e2a-4465-8825-ab1a1d82dada" containerName="kube-rbac-proxy" Oct 30 11:35:36.952187 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952177 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c23b2839-9e2a-4465-8825-ab1a1d82dada" containerName="kube-rbac-proxy" Oct 30 11:35:36.952187 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952188 4289 cpu_manager.go:414] "containerMap: removing container" podUID="85f4e48a-f1de-4fe7-a113-c096686d57bb" containerName="otc-container" Oct 30 11:35:36.952369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952193 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f4e48a-f1de-4fe7-a113-c096686d57bb" containerName="otc-container" Oct 30 11:35:36.952369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952202 4289 cpu_manager.go:414] "containerMap: removing container" podUID="708f844f-1c03-4e6f-95b9-7f3b8dabff06" containerName="method-gate" Oct 30 11:35:36.952369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952206 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708f844f-1c03-4e6f-95b9-7f3b8dabff06" containerName="method-gate" Oct 30 11:35:36.952369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952214 4289 cpu_manager.go:414] "containerMap: removing container" podUID="708f844f-1c03-4e6f-95b9-7f3b8dabff06" containerName="prom-label-proxy" Oct 30 11:35:36.952369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952218 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708f844f-1c03-4e6f-95b9-7f3b8dabff06" containerName="prom-label-proxy" Oct 30 11:35:36.952369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952226 4289 cpu_manager.go:414] "containerMap: removing container" podUID="c23b2839-9e2a-4465-8825-ab1a1d82dada" containerName="prom-label-proxy" Oct 30 11:35:36.952369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952230 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c23b2839-9e2a-4465-8825-ab1a1d82dada" containerName="prom-label-proxy" Oct 30 11:35:36.952369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952237 4289 cpu_manager.go:414] "containerMap: removing container" podUID="708f844f-1c03-4e6f-95b9-7f3b8dabff06" containerName="kube-rbac-proxy" Oct 30 11:35:36.952369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952241 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708f844f-1c03-4e6f-95b9-7f3b8dabff06" containerName="kube-rbac-proxy" Oct 30 11:35:36.952369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952248 4289 cpu_manager.go:414] "containerMap: removing container" podUID="c23b2839-9e2a-4465-8825-ab1a1d82dada" containerName="method-gate" Oct 30 11:35:36.952369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952252 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c23b2839-9e2a-4465-8825-ab1a1d82dada" containerName="method-gate" Oct 30 11:35:36.952369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952301 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="c23b2839-9e2a-4465-8825-ab1a1d82dada" containerName="method-gate" Oct 30 11:35:36.952369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952311 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff9592ba-475a-4ced-b3a2-92694daf0e15" containerName="korrel8r" Oct 30 11:35:36.952369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952320 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1a60678-72d4-453c-8c02-ea030952c1ce" containerName="kube-rbac-proxy" Oct 30 11:35:36.952369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952326 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="c23b2839-9e2a-4465-8825-ab1a1d82dada" containerName="kube-rbac-proxy" Oct 30 11:35:36.952369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952332 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f4e48a-f1de-4fe7-a113-c096686d57bb" containerName="otc-container" Oct 30 11:35:36.952369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952337 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="708f844f-1c03-4e6f-95b9-7f3b8dabff06" containerName="method-gate" Oct 30 11:35:36.952369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952342 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="708f844f-1c03-4e6f-95b9-7f3b8dabff06" containerName="prom-label-proxy" Oct 30 11:35:36.952369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952348 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="c23b2839-9e2a-4465-8825-ab1a1d82dada" containerName="prom-label-proxy" Oct 30 11:35:36.952369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.952353 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="708f844f-1c03-4e6f-95b9-7f3b8dabff06" containerName="kube-rbac-proxy" Oct 30 11:35:36.955389 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.955372 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:35:36.970418 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.970396 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 11:35:36.970590 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.970450 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 11:35:36.970590 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.970508 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 11:35:36.970590 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.970567 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-e4cadd38\"" Oct 30 11:35:36.970698 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.970631 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-7p6tb\"" Oct 30 11:35:36.976154 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:36.976128 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 11:35:37.022836 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.022804 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:37.044965 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.044933 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:35:37.045118 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.044978 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:35:37.045118 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.045097 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:35:37.045195 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.045125 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6phpm\" (UniqueName: \"kubernetes.io/projected/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-kube-api-access-6phpm\") pod \"data-science-collector-collector-0\" (UID: \"8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:35:37.045195 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.045173 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:35:37.145878 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.145784 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:35:37.145878 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.145844 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:35:37.145878 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.145870 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6phpm\" (UniqueName: \"kubernetes.io/projected/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-kube-api-access-6phpm\") pod \"data-science-collector-collector-0\" (UID: \"8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:35:37.146159 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.145888 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:35:37.146159 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.145961 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:35:37.146645 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.146617 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:35:37.148732 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.148705 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:35:37.148863 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.148840 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:35:37.149231 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.149213 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:35:37.160769 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:35:37.160742 4289 projected.go:196] Error preparing data for projected volume kube-api-access-6phpm for pod redhat-ods-monitoring/data-science-collector-collector-0: failed to fetch token: serviceaccounts "data-science-collector-collector" not found Oct 30 11:35:37.160931 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:35:37.160823 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-kube-api-access-6phpm podName:8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d nodeName:}" failed. No retries permitted until 2026-10-30 11:35:37.660800952 +0000 UTC m=+3028.162180427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6phpm" (UniqueName: "kubernetes.io/projected/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-kube-api-access-6phpm") pod "data-science-collector-collector-0" (UID: "8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d") : failed to fetch token: serviceaccounts "data-science-collector-collector" not found Oct 30 11:35:37.184176 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.184142 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 11:35:37.184498 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:35:37.184473 4289 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-6phpm], unattached volumes=[], failed to process volumes=[]: context canceled" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d" Oct 30 11:35:37.218020 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.217978 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.132:9093/-/ready\": dial tcp 10.134.0.132:9093: connect: connection refused" Oct 30 11:35:37.260761 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.260720 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:37.357993 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.357966 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:37.358609 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.358590 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" Oct 30 11:35:37.358609 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.358597 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6" event={"ID":"c1a60678-72d4-453c-8c02-ea030952c1ce","Type":"ContainerDied","Data":"72842edaeb90b7e794b13e27f439fcf293c6a496091ff2167722018c910e74e8"} Oct 30 11:35:37.358809 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.358658 4289 scope.go:122] "RemoveContainer" containerID="4fd04b49bb8dfae30add8b4d4108a714d1c5910f04a6f7e19a3fe4110ee2c8c0" Oct 30 11:35:37.361032 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.361009 4289 generic.go:356] "Generic (PLEG): container finished" podID="07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1" containerID="83bddfda9583cdc7934fb3ad3855da07edffc3849260129d131867e78ae7d9a7" exitCode=0 Oct 30 11:35:37.361133 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.361080 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1","Type":"ContainerDied","Data":"83bddfda9583cdc7934fb3ad3855da07edffc3849260129d131867e78ae7d9a7"} Oct 30 11:35:37.363194 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.363167 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" event={"ID":"c23b2839-9e2a-4465-8825-ab1a1d82dada","Type":"ContainerDied","Data":"8c78399eb93437b784ef653ae1ca3e197f5f577594ba1b77f6f5e167fc54cdf5"} Oct 30 11:35:37.363303 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.363171 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd" Oct 30 11:35:37.365223 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.365205 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" Oct 30 11:35:37.365327 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.365222 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz" event={"ID":"708f844f-1c03-4e6f-95b9-7f3b8dabff06","Type":"ContainerDied","Data":"71dda2a28a3783017de09368a57d579076cca788dfadd272b312986914a8d303"} Oct 30 11:35:37.367855 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.367235 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz" event={"ID":"ff9592ba-475a-4ced-b3a2-92694daf0e15","Type":"ContainerDied","Data":"7570a613aa15af2a1d59bf425557655020c0c17bce49af2a99a4140600e4df82"} Oct 30 11:35:37.367855 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.367243 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz" Oct 30 11:35:37.373819 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.373725 4289 scope.go:122] "RemoveContainer" containerID="a3dd94f4c2a4581caf4c608afc5aad0fe0f7c8c5b9fc0680a59572c7f18aec2c" Oct 30 11:35:37.376749 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.376730 4289 generic.go:356] "Generic (PLEG): container finished" podID="b98eecff-d807-4b6c-a02e-d85763da9485" containerID="958b8b883b30e675c4d8cb9795253c467703e49b36bec654f2172a7c17e40ac7" exitCode=0 Oct 30 11:35:37.376860 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.376751 4289 generic.go:356] "Generic (PLEG): container finished" podID="b98eecff-d807-4b6c-a02e-d85763da9485" containerID="a7e015a68cf0961b3b10edaeaf3735346099c35b12725573dca1e9637014c666" exitCode=0 Oct 30 11:35:37.376860 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.376760 4289 generic.go:356] "Generic (PLEG): container finished" podID="b98eecff-d807-4b6c-a02e-d85763da9485" containerID="229cec0536cb47ce8f03f713a5977fc603a9aea9bd04a81c3125a8eb19831d2d" exitCode=0 Oct 30 11:35:37.376860 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.376794 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:37.376860 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.376827 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b98eecff-d807-4b6c-a02e-d85763da9485","Type":"ContainerDied","Data":"958b8b883b30e675c4d8cb9795253c467703e49b36bec654f2172a7c17e40ac7"} Oct 30 11:35:37.377138 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.376885 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b98eecff-d807-4b6c-a02e-d85763da9485","Type":"ContainerDied","Data":"a7e015a68cf0961b3b10edaeaf3735346099c35b12725573dca1e9637014c666"} Oct 30 11:35:37.377138 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.376926 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b98eecff-d807-4b6c-a02e-d85763da9485","Type":"ContainerDied","Data":"229cec0536cb47ce8f03f713a5977fc603a9aea9bd04a81c3125a8eb19831d2d"} Oct 30 11:35:37.377138 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.376964 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b98eecff-d807-4b6c-a02e-d85763da9485","Type":"ContainerDied","Data":"8e358cc3166cab4bc85e2411b8ed3a0620606b483ead647e4db0fa85e166a045"} Oct 30 11:35:37.377138 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.377117 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:35:37.385233 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.385214 4289 scope.go:122] "RemoveContainer" containerID="00276833e29815457b8fdea7aaa0d696f658648f35ffee255d8f7e9becbf8906" Oct 30 11:35:37.387920 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.387901 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:35:37.392896 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.392880 4289 scope.go:122] "RemoveContainer" containerID="19e0f03fa4e00667e1cca4bf102c8abb1ca96f9b0e5b1677f8adac76e4954ec6" Oct 30 11:35:37.399871 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.399855 4289 scope.go:122] "RemoveContainer" containerID="bf7e5ead5ca9a3126391e7bba5b706e9c922675b9aec21a900e9a0633c26ba64" Oct 30 11:35:37.406748 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.406729 4289 scope.go:122] "RemoveContainer" containerID="24698e8db6f2a06f965ed6cbcc4dfd67ef3c0ad979c3a0b304d243665277957e" Oct 30 11:35:37.414316 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.414292 4289 scope.go:122] "RemoveContainer" containerID="ec4ea8bc43eef93b09798857da7657d14e597d594cfcfcff6f7d3b515033bd43" Oct 30 11:35:37.422331 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.422309 4289 scope.go:122] "RemoveContainer" containerID="428f96ce2cee38489f5882fb90d1a118956a227929f67ee5291ee9c3e3bb1805" Oct 30 11:35:37.430901 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.430876 4289 scope.go:122] "RemoveContainer" containerID="958b8b883b30e675c4d8cb9795253c467703e49b36bec654f2172a7c17e40ac7" Oct 30 11:35:37.432873 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.432847 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:37.443487 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.443468 4289 scope.go:122] "RemoveContainer" containerID="a7e015a68cf0961b3b10edaeaf3735346099c35b12725573dca1e9637014c666" Oct 30 11:35:37.447609 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.447584 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd"] Oct 30 11:35:37.448494 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.448474 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-config\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-config\") pod \"b98eecff-d807-4b6c-a02e-d85763da9485\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " Oct 30 11:35:37.448684 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.448662 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-thanos-prometheus-http-client-file\") pod \"b98eecff-d807-4b6c-a02e-d85763da9485\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " Oct 30 11:35:37.449052 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.449028 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-monitoring-tls-certs\") pod \"8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d\" (UID: \"8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d\") " Oct 30 11:35:37.449052 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.449069 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b98eecff-d807-4b6c-a02e-d85763da9485-kube-api-access-wk4dg\" (UniqueName: \"kubernetes.io/projected/b98eecff-d807-4b6c-a02e-d85763da9485-kube-api-access-wk4dg\") pod \"b98eecff-d807-4b6c-a02e-d85763da9485\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " Oct 30 11:35:37.449052 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.449109 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-secret-prometheus-web-tls-ca\") pod \"b98eecff-d807-4b6c-a02e-d85763da9485\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " Oct 30 11:35:37.449052 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.449206 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"b98eecff-d807-4b6c-a02e-d85763da9485\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " Oct 30 11:35:37.449052 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.449237 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-web-config\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-web-config\") pod \"b98eecff-d807-4b6c-a02e-d85763da9485\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " Oct 30 11:35:37.449719 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.449273 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"b98eecff-d807-4b6c-a02e-d85763da9485\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " Oct 30 11:35:37.449719 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.449306 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"b98eecff-d807-4b6c-a02e-d85763da9485\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " Oct 30 11:35:37.449719 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.449329 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b98eecff-d807-4b6c-a02e-d85763da9485-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b98eecff-d807-4b6c-a02e-d85763da9485-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"b98eecff-d807-4b6c-a02e-d85763da9485\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " Oct 30 11:35:37.449719 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.449348 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b98eecff-d807-4b6c-a02e-d85763da9485-config-out\" (UniqueName: \"kubernetes.io/empty-dir/b98eecff-d807-4b6c-a02e-d85763da9485-config-out\") pod \"b98eecff-d807-4b6c-a02e-d85763da9485\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " Oct 30 11:35:37.449719 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.449366 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b98eecff-d807-4b6c-a02e-d85763da9485-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b98eecff-d807-4b6c-a02e-d85763da9485-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"b98eecff-d807-4b6c-a02e-d85763da9485\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " Oct 30 11:35:37.449719 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.449391 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-tls-certs\") pod \"8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d\" (UID: \"8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d\") " Oct 30 11:35:37.449719 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.449459 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-data-science-collector-ta-client-cert\") pod \"8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d\" (UID: \"8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d\") " Oct 30 11:35:37.449719 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.449503 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b98eecff-d807-4b6c-a02e-d85763da9485-tls-assets\" (UniqueName: \"kubernetes.io/projected/b98eecff-d807-4b6c-a02e-d85763da9485-tls-assets\") pod \"b98eecff-d807-4b6c-a02e-d85763da9485\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " Oct 30 11:35:37.449719 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.449542 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-otc-internal\" (UniqueName: \"kubernetes.io/configmap/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-otc-internal\") pod \"8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d\" (UID: \"8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d\") " Oct 30 11:35:37.449719 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.449562 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b98eecff-d807-4b6c-a02e-d85763da9485-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b98eecff-d807-4b6c-a02e-d85763da9485-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"b98eecff-d807-4b6c-a02e-d85763da9485\" (UID: \"b98eecff-d807-4b6c-a02e-d85763da9485\") " Oct 30 11:35:37.450249 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.450051 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b98eecff-d807-4b6c-a02e-d85763da9485-prometheus-data-science-monitoringstack-rulefiles-2" pod "b98eecff-d807-4b6c-a02e-d85763da9485" (UID: "b98eecff-d807-4b6c-a02e-d85763da9485"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:35:37.450984 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.450670 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b98eecff-d807-4b6c-a02e-d85763da9485-prometheus-data-science-monitoringstack-rulefiles-0" pod "b98eecff-d807-4b6c-a02e-d85763da9485" (UID: "b98eecff-d807-4b6c-a02e-d85763da9485"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:35:37.452121 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.451858 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-otc-internal" pod "8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d" (UID: "8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:35:37.454442 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.454187 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b98eecff-d807-4b6c-a02e-d85763da9485-prometheus-data-science-monitoringstack-rulefiles-1" pod "b98eecff-d807-4b6c-a02e-d85763da9485" (UID: "b98eecff-d807-4b6c-a02e-d85763da9485"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:35:37.455296 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.454711 4289 scope.go:122] "RemoveContainer" containerID="229cec0536cb47ce8f03f713a5977fc603a9aea9bd04a81c3125a8eb19831d2d" Oct 30 11:35:37.456200 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.456143 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-79scd"] Oct 30 11:35:37.457787 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.457752 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-secret-prometheus-web-tls-ca" pod "b98eecff-d807-4b6c-a02e-d85763da9485" (UID: "b98eecff-d807-4b6c-a02e-d85763da9485"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:35:37.458124 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.458103 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-tls-certs" pod "8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d" (UID: "8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:35:37.458263 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.458109 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "b98eecff-d807-4b6c-a02e-d85763da9485" (UID: "b98eecff-d807-4b6c-a02e-d85763da9485"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:35:37.458263 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.458252 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98eecff-d807-4b6c-a02e-d85763da9485-kube-api-access-wk4dg" pod "b98eecff-d807-4b6c-a02e-d85763da9485" (UID: "b98eecff-d807-4b6c-a02e-d85763da9485"). InnerVolumeSpecName "kube-api-access-wk4dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:35:37.458405 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.458276 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98eecff-d807-4b6c-a02e-d85763da9485-tls-assets" pod "b98eecff-d807-4b6c-a02e-d85763da9485" (UID: "b98eecff-d807-4b6c-a02e-d85763da9485"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:35:37.458719 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.458695 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-monitoring-tls-certs" pod "8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d" (UID: "8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:35:37.458837 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.458715 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-data-science-collector-ta-client-cert" pod "8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d" (UID: "8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:35:37.458837 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.458799 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-thanos-prometheus-http-client-file" pod "b98eecff-d807-4b6c-a02e-d85763da9485" (UID: "b98eecff-d807-4b6c-a02e-d85763da9485"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:35:37.458837 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.458814 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-config" pod "b98eecff-d807-4b6c-a02e-d85763da9485" (UID: "b98eecff-d807-4b6c-a02e-d85763da9485"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:35:37.459422 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.459379 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "b98eecff-d807-4b6c-a02e-d85763da9485" (UID: "b98eecff-d807-4b6c-a02e-d85763da9485"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:35:37.460504 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.460478 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b98eecff-d807-4b6c-a02e-d85763da9485-config-out" pod "b98eecff-d807-4b6c-a02e-d85763da9485" (UID: "b98eecff-d807-4b6c-a02e-d85763da9485"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:35:37.462822 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.462794 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48" pod "b98eecff-d807-4b6c-a02e-d85763da9485" (UID: "b98eecff-d807-4b6c-a02e-d85763da9485"). InnerVolumeSpecName "pvc-c799cc48-125a-421c-8fc6-8a4d017a7653". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 11:35:37.464676 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.464656 4289 scope.go:122] "RemoveContainer" containerID="7b89ddd71cfe1f630782a4de1d6037f59c5f52f8ac53800125eaeb7ce7244cbd" Oct 30 11:35:37.472206 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.472186 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-web-config" pod "b98eecff-d807-4b6c-a02e-d85763da9485" (UID: "b98eecff-d807-4b6c-a02e-d85763da9485"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:35:37.473133 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.473115 4289 scope.go:122] "RemoveContainer" containerID="958b8b883b30e675c4d8cb9795253c467703e49b36bec654f2172a7c17e40ac7" Oct 30 11:35:37.473407 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:35:37.473389 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"958b8b883b30e675c4d8cb9795253c467703e49b36bec654f2172a7c17e40ac7\": container with ID starting with 958b8b883b30e675c4d8cb9795253c467703e49b36bec654f2172a7c17e40ac7 not found: ID does not exist" containerID="958b8b883b30e675c4d8cb9795253c467703e49b36bec654f2172a7c17e40ac7" Oct 30 11:35:37.473628 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.473414 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"958b8b883b30e675c4d8cb9795253c467703e49b36bec654f2172a7c17e40ac7"} err="failed to get container status \"958b8b883b30e675c4d8cb9795253c467703e49b36bec654f2172a7c17e40ac7\": rpc error: code = NotFound desc = could not find container \"958b8b883b30e675c4d8cb9795253c467703e49b36bec654f2172a7c17e40ac7\": container with ID starting with 958b8b883b30e675c4d8cb9795253c467703e49b36bec654f2172a7c17e40ac7 not found: ID does not exist" Oct 30 11:35:37.473628 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.473447 4289 scope.go:122] "RemoveContainer" containerID="a7e015a68cf0961b3b10edaeaf3735346099c35b12725573dca1e9637014c666" Oct 30 11:35:37.473761 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:35:37.473713 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7e015a68cf0961b3b10edaeaf3735346099c35b12725573dca1e9637014c666\": container with ID starting with a7e015a68cf0961b3b10edaeaf3735346099c35b12725573dca1e9637014c666 not found: ID does not exist" containerID="a7e015a68cf0961b3b10edaeaf3735346099c35b12725573dca1e9637014c666" Oct 30 11:35:37.473761 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.473745 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7e015a68cf0961b3b10edaeaf3735346099c35b12725573dca1e9637014c666"} err="failed to get container status \"a7e015a68cf0961b3b10edaeaf3735346099c35b12725573dca1e9637014c666\": rpc error: code = NotFound desc = could not find container \"a7e015a68cf0961b3b10edaeaf3735346099c35b12725573dca1e9637014c666\": container with ID starting with a7e015a68cf0961b3b10edaeaf3735346099c35b12725573dca1e9637014c666 not found: ID does not exist" Oct 30 11:35:37.473871 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.473768 4289 scope.go:122] "RemoveContainer" containerID="229cec0536cb47ce8f03f713a5977fc603a9aea9bd04a81c3125a8eb19831d2d" Oct 30 11:35:37.474013 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:35:37.473997 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"229cec0536cb47ce8f03f713a5977fc603a9aea9bd04a81c3125a8eb19831d2d\": container with ID starting with 229cec0536cb47ce8f03f713a5977fc603a9aea9bd04a81c3125a8eb19831d2d not found: ID does not exist" containerID="229cec0536cb47ce8f03f713a5977fc603a9aea9bd04a81c3125a8eb19831d2d" Oct 30 11:35:37.474098 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.474021 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"229cec0536cb47ce8f03f713a5977fc603a9aea9bd04a81c3125a8eb19831d2d"} err="failed to get container status \"229cec0536cb47ce8f03f713a5977fc603a9aea9bd04a81c3125a8eb19831d2d\": rpc error: code = NotFound desc = could not find container \"229cec0536cb47ce8f03f713a5977fc603a9aea9bd04a81c3125a8eb19831d2d\": container with ID starting with 229cec0536cb47ce8f03f713a5977fc603a9aea9bd04a81c3125a8eb19831d2d not found: ID does not exist" Oct 30 11:35:37.474098 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.474038 4289 scope.go:122] "RemoveContainer" containerID="7b89ddd71cfe1f630782a4de1d6037f59c5f52f8ac53800125eaeb7ce7244cbd" Oct 30 11:35:37.474271 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:35:37.474251 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b89ddd71cfe1f630782a4de1d6037f59c5f52f8ac53800125eaeb7ce7244cbd\": container with ID starting with 7b89ddd71cfe1f630782a4de1d6037f59c5f52f8ac53800125eaeb7ce7244cbd not found: ID does not exist" containerID="7b89ddd71cfe1f630782a4de1d6037f59c5f52f8ac53800125eaeb7ce7244cbd" Oct 30 11:35:37.474343 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.474277 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b89ddd71cfe1f630782a4de1d6037f59c5f52f8ac53800125eaeb7ce7244cbd"} err="failed to get container status \"7b89ddd71cfe1f630782a4de1d6037f59c5f52f8ac53800125eaeb7ce7244cbd\": rpc error: code = NotFound desc = could not find container \"7b89ddd71cfe1f630782a4de1d6037f59c5f52f8ac53800125eaeb7ce7244cbd\": container with ID starting with 7b89ddd71cfe1f630782a4de1d6037f59c5f52f8ac53800125eaeb7ce7244cbd not found: ID does not exist" Oct 30 11:35:37.474343 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.474296 4289 scope.go:122] "RemoveContainer" containerID="958b8b883b30e675c4d8cb9795253c467703e49b36bec654f2172a7c17e40ac7" Oct 30 11:35:37.474610 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.474593 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"958b8b883b30e675c4d8cb9795253c467703e49b36bec654f2172a7c17e40ac7"} err="failed to get container status \"958b8b883b30e675c4d8cb9795253c467703e49b36bec654f2172a7c17e40ac7\": rpc error: code = NotFound desc = could not find container \"958b8b883b30e675c4d8cb9795253c467703e49b36bec654f2172a7c17e40ac7\": container with ID starting with 958b8b883b30e675c4d8cb9795253c467703e49b36bec654f2172a7c17e40ac7 not found: ID does not exist" Oct 30 11:35:37.474682 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.474611 4289 scope.go:122] "RemoveContainer" containerID="a7e015a68cf0961b3b10edaeaf3735346099c35b12725573dca1e9637014c666" Oct 30 11:35:37.474870 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.474849 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7e015a68cf0961b3b10edaeaf3735346099c35b12725573dca1e9637014c666"} err="failed to get container status \"a7e015a68cf0961b3b10edaeaf3735346099c35b12725573dca1e9637014c666\": rpc error: code = NotFound desc = could not find container \"a7e015a68cf0961b3b10edaeaf3735346099c35b12725573dca1e9637014c666\": container with ID starting with a7e015a68cf0961b3b10edaeaf3735346099c35b12725573dca1e9637014c666 not found: ID does not exist" Oct 30 11:35:37.474925 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.474870 4289 scope.go:122] "RemoveContainer" containerID="229cec0536cb47ce8f03f713a5977fc603a9aea9bd04a81c3125a8eb19831d2d" Oct 30 11:35:37.475088 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.475071 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"229cec0536cb47ce8f03f713a5977fc603a9aea9bd04a81c3125a8eb19831d2d"} err="failed to get container status \"229cec0536cb47ce8f03f713a5977fc603a9aea9bd04a81c3125a8eb19831d2d\": rpc error: code = NotFound desc = could not find container \"229cec0536cb47ce8f03f713a5977fc603a9aea9bd04a81c3125a8eb19831d2d\": container with ID starting with 229cec0536cb47ce8f03f713a5977fc603a9aea9bd04a81c3125a8eb19831d2d not found: ID does not exist" Oct 30 11:35:37.475138 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.475088 4289 scope.go:122] "RemoveContainer" containerID="7b89ddd71cfe1f630782a4de1d6037f59c5f52f8ac53800125eaeb7ce7244cbd" Oct 30 11:35:37.475277 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.475255 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b89ddd71cfe1f630782a4de1d6037f59c5f52f8ac53800125eaeb7ce7244cbd"} err="failed to get container status \"7b89ddd71cfe1f630782a4de1d6037f59c5f52f8ac53800125eaeb7ce7244cbd\": rpc error: code = NotFound desc = could not find container \"7b89ddd71cfe1f630782a4de1d6037f59c5f52f8ac53800125eaeb7ce7244cbd\": container with ID starting with 7b89ddd71cfe1f630782a4de1d6037f59c5f52f8ac53800125eaeb7ce7244cbd not found: ID does not exist" Oct 30 11:35:37.475351 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.475278 4289 scope.go:122] "RemoveContainer" containerID="958b8b883b30e675c4d8cb9795253c467703e49b36bec654f2172a7c17e40ac7" Oct 30 11:35:37.475541 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.475522 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"958b8b883b30e675c4d8cb9795253c467703e49b36bec654f2172a7c17e40ac7"} err="failed to get container status \"958b8b883b30e675c4d8cb9795253c467703e49b36bec654f2172a7c17e40ac7\": rpc error: code = NotFound desc = could not find container \"958b8b883b30e675c4d8cb9795253c467703e49b36bec654f2172a7c17e40ac7\": container with ID starting with 958b8b883b30e675c4d8cb9795253c467703e49b36bec654f2172a7c17e40ac7 not found: ID does not exist" Oct 30 11:35:37.475541 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.475541 4289 scope.go:122] "RemoveContainer" containerID="a7e015a68cf0961b3b10edaeaf3735346099c35b12725573dca1e9637014c666" Oct 30 11:35:37.475805 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.475787 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7e015a68cf0961b3b10edaeaf3735346099c35b12725573dca1e9637014c666"} err="failed to get container status \"a7e015a68cf0961b3b10edaeaf3735346099c35b12725573dca1e9637014c666\": rpc error: code = NotFound desc = could not find container \"a7e015a68cf0961b3b10edaeaf3735346099c35b12725573dca1e9637014c666\": container with ID starting with a7e015a68cf0961b3b10edaeaf3735346099c35b12725573dca1e9637014c666 not found: ID does not exist" Oct 30 11:35:37.475892 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.475817 4289 scope.go:122] "RemoveContainer" containerID="229cec0536cb47ce8f03f713a5977fc603a9aea9bd04a81c3125a8eb19831d2d" Oct 30 11:35:37.476046 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.476024 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"229cec0536cb47ce8f03f713a5977fc603a9aea9bd04a81c3125a8eb19831d2d"} err="failed to get container status \"229cec0536cb47ce8f03f713a5977fc603a9aea9bd04a81c3125a8eb19831d2d\": rpc error: code = NotFound desc = could not find container \"229cec0536cb47ce8f03f713a5977fc603a9aea9bd04a81c3125a8eb19831d2d\": container with ID starting with 229cec0536cb47ce8f03f713a5977fc603a9aea9bd04a81c3125a8eb19831d2d not found: ID does not exist" Oct 30 11:35:37.476099 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.476047 4289 scope.go:122] "RemoveContainer" containerID="7b89ddd71cfe1f630782a4de1d6037f59c5f52f8ac53800125eaeb7ce7244cbd" Oct 30 11:35:37.476263 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.476240 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b89ddd71cfe1f630782a4de1d6037f59c5f52f8ac53800125eaeb7ce7244cbd"} err="failed to get container status \"7b89ddd71cfe1f630782a4de1d6037f59c5f52f8ac53800125eaeb7ce7244cbd\": rpc error: code = NotFound desc = could not find container \"7b89ddd71cfe1f630782a4de1d6037f59c5f52f8ac53800125eaeb7ce7244cbd\": container with ID starting with 7b89ddd71cfe1f630782a4de1d6037f59c5f52f8ac53800125eaeb7ce7244cbd not found: ID does not exist" Oct 30 11:35:37.488371 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.488348 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz"] Oct 30 11:35:37.494705 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.494684 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-kztxz"] Oct 30 11:35:37.527581 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.527554 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz"] Oct 30 11:35:37.531713 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.531689 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-gj5lz"] Oct 30 11:35:37.550881 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.550857 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b98eecff-d807-4b6c-a02e-d85763da9485-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:37.550981 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.550887 4289 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b98eecff-d807-4b6c-a02e-d85763da9485-config-out\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:37.550981 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.550898 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b98eecff-d807-4b6c-a02e-d85763da9485-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:37.550981 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.550908 4289 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:37.550981 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.550918 4289 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-data-science-collector-ta-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:37.550981 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.550927 4289 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b98eecff-d807-4b6c-a02e-d85763da9485-tls-assets\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:37.550981 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.550935 4289 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-otc-internal\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:37.550981 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.550943 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b98eecff-d807-4b6c-a02e-d85763da9485-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:37.550981 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.550951 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:37.550981 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.550959 4289 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:37.550981 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.550967 4289 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-monitoring-tls-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:37.550981 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.550976 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk4dg\" (UniqueName: \"kubernetes.io/projected/b98eecff-d807-4b6c-a02e-d85763da9485-kube-api-access-wk4dg\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:37.550981 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.550984 4289 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-secret-prometheus-web-tls-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:37.551554 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.551007 4289 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") on node \"ip-10-0-137-242.ec2.internal\" " Oct 30 11:35:37.551554 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.551016 4289 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-web-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:37.551554 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.551025 4289 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:37.551554 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.551033 4289 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b98eecff-d807-4b6c-a02e-d85763da9485-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:37.561915 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.561900 4289 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-c799cc48-125a-421c-8fc6-8a4d017a7653" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48") on node "ip-10-0-137-242.ec2.internal" Oct 30 11:35:37.580413 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.580386 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6"] Oct 30 11:35:37.584087 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.584063 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-gnxk6"] Oct 30 11:35:37.603680 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.603636 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:37.625217 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.625188 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="708f844f-1c03-4e6f-95b9-7f3b8dabff06" path="/var/lib/kubelet/pods/708f844f-1c03-4e6f-95b9-7f3b8dabff06/volumes" Oct 30 11:35:37.625678 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.625660 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1a60678-72d4-453c-8c02-ea030952c1ce" path="/var/lib/kubelet/pods/c1a60678-72d4-453c-8c02-ea030952c1ce/volumes" Oct 30 11:35:37.626002 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.625988 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c23b2839-9e2a-4465-8825-ab1a1d82dada" path="/var/lib/kubelet/pods/c23b2839-9e2a-4465-8825-ab1a1d82dada/volumes" Oct 30 11:35:37.626736 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.626719 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff9592ba-475a-4ced-b3a2-92694daf0e15" path="/var/lib/kubelet/pods/ff9592ba-475a-4ced-b3a2-92694daf0e15/volumes" Oct 30 11:35:37.652253 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.652192 4289 reconciler_common.go:299] "Volume detached for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"csi-ea06c66241a0a003a5daff5d9fc602f40e53768b00b104c5e4118b92d558c090\"" Oct 30 11:35:37.716206 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.716174 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:35:37.722137 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.722107 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:35:37.752932 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:37.752896 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6phpm\" (UniqueName: \"kubernetes.io/projected/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-kube-api-access-6phpm\") pod \"data-science-collector-collector-0\" (UID: \"8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:35:37.767541 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:35:37.767504 4289 projected.go:196] Error preparing data for projected volume kube-api-access-6phpm for pod redhat-ods-monitoring/data-science-collector-collector-0: failed to fetch token: serviceaccounts "data-science-collector-collector" not found Oct 30 11:35:37.767695 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:35:37.767587 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-kube-api-access-6phpm podName:8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d nodeName:}" failed. No retries permitted until 2026-10-30 11:35:38.767566902 +0000 UTC m=+3029.268946378 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6phpm" (UniqueName: "kubernetes.io/projected/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-kube-api-access-6phpm") pod "data-science-collector-collector-0" (UID: "8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d") : failed to fetch token: serviceaccounts "data-science-collector-collector" not found Oct 30 11:35:38.151720 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:38.151647 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:38.317947 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:38.317852 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:38.386262 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:38.386227 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 11:35:38.461539 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:38.461509 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 11:35:38.464746 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:38.464716 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 11:35:38.482129 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:38.482103 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:38.559744 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:38.559714 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6phpm\" (UniqueName: \"kubernetes.io/projected/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d-kube-api-access-6phpm\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:39.459607 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:39.459572 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:39.625041 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:39.625007 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d" path="/var/lib/kubelet/pods/8a1e4c65-55e4-4d7e-a86e-cd8d4e72632d/volumes" Oct 30 11:35:39.625361 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:39.625348 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b98eecff-d807-4b6c-a02e-d85763da9485" path="/var/lib/kubelet/pods/b98eecff-d807-4b6c-a02e-d85763da9485/volumes" Oct 30 11:35:42.217512 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:42.217466 4289 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.132:9093/-/ready\": dial tcp 10.134.0.132:9093: connect: connection refused" Oct 30 11:35:44.404277 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.404236 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:44.589224 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.589192 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:44.658795 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.658714 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc"] Oct 30 11:35:44.659046 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.659034 4289 cpu_manager.go:414] "containerMap: removing container" podUID="b98eecff-d807-4b6c-a02e-d85763da9485" containerName="thanos-sidecar" Oct 30 11:35:44.659084 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.659047 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b98eecff-d807-4b6c-a02e-d85763da9485" containerName="thanos-sidecar" Oct 30 11:35:44.659084 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.659070 4289 cpu_manager.go:414] "containerMap: removing container" podUID="b98eecff-d807-4b6c-a02e-d85763da9485" containerName="prometheus" Oct 30 11:35:44.659084 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.659075 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b98eecff-d807-4b6c-a02e-d85763da9485" containerName="prometheus" Oct 30 11:35:44.659084 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.659081 4289 cpu_manager.go:414] "containerMap: removing container" podUID="b98eecff-d807-4b6c-a02e-d85763da9485" containerName="config-reloader" Oct 30 11:35:44.659213 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.659086 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b98eecff-d807-4b6c-a02e-d85763da9485" containerName="config-reloader" Oct 30 11:35:44.659213 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.659096 4289 cpu_manager.go:414] "containerMap: removing container" podUID="b98eecff-d807-4b6c-a02e-d85763da9485" containerName="init-config-reloader" Oct 30 11:35:44.659213 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.659101 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b98eecff-d807-4b6c-a02e-d85763da9485" containerName="init-config-reloader" Oct 30 11:35:44.659213 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.659150 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="b98eecff-d807-4b6c-a02e-d85763da9485" containerName="prometheus" Oct 30 11:35:44.659213 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.659158 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="b98eecff-d807-4b6c-a02e-d85763da9485" containerName="config-reloader" Oct 30 11:35:44.659213 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.659167 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="b98eecff-d807-4b6c-a02e-d85763da9485" containerName="thanos-sidecar" Oct 30 11:35:44.662116 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.662101 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" Oct 30 11:35:44.689177 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.689128 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 11:35:44.689177 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.689150 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 11:35:44.689177 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.689155 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 11:35:44.689481 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.689185 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 11:35:44.690584 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.690560 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc"] Oct 30 11:35:44.691595 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.691577 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-rszd6\"" Oct 30 11:35:44.699940 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.699919 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq"] Oct 30 11:35:44.703401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.703386 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" Oct 30 11:35:44.727661 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.727640 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq"] Oct 30 11:35:44.761592 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.761562 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq"] Oct 30 11:35:44.764983 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.764964 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" Oct 30 11:35:44.780046 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.780018 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 11:35:44.780152 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.780072 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 11:35:44.780346 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.780329 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-9rnpn\"" Oct 30 11:35:44.780419 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.780405 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 11:35:44.782190 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.782172 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-bslg8\"" Oct 30 11:35:44.782301 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.782189 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 11:35:44.782393 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.782379 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 11:35:44.788031 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.788007 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq"] Oct 30 11:35:44.795404 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.795380 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk"] Oct 30 11:35:44.798096 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.798074 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:44.798772 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.798759 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk" Oct 30 11:35:44.815919 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.815897 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/610b3ae0-322e-4bba-bd8c-22b37b0afd25-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq\" (UID: \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" Oct 30 11:35:44.816044 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.815938 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m42hk\" (UniqueName: \"kubernetes.io/projected/610b3ae0-322e-4bba-bd8c-22b37b0afd25-kube-api-access-m42hk\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq\" (UID: \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" Oct 30 11:35:44.816044 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.816020 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-swgqc\" (UID: \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" Oct 30 11:35:44.816163 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.816091 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/610b3ae0-322e-4bba-bd8c-22b37b0afd25-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq\" (UID: \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" Oct 30 11:35:44.816163 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.816131 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/610b3ae0-322e-4bba-bd8c-22b37b0afd25-config\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq\" (UID: \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" Oct 30 11:35:44.816255 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.816194 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-swgqc\" (UID: \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" Oct 30 11:35:44.816255 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.816241 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-swgqc\" (UID: \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" Oct 30 11:35:44.816315 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.816262 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-swgqc\" (UID: \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" Oct 30 11:35:44.816315 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.816280 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/610b3ae0-322e-4bba-bd8c-22b37b0afd25-client-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq\" (UID: \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" Oct 30 11:35:44.816315 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.816296 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc4gp\" (UniqueName: \"kubernetes.io/projected/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-kube-api-access-lc4gp\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-swgqc\" (UID: \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" Oct 30 11:35:44.824588 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.824566 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk"] Oct 30 11:35:44.881524 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.881491 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 11:35:44.881524 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.881523 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-k5bvx\"" Oct 30 11:35:44.881744 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.881561 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 11:35:44.916885 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.916814 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/610b3ae0-322e-4bba-bd8c-22b37b0afd25-config\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq\" (UID: \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" Oct 30 11:35:44.916885 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.916855 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-swgqc\" (UID: \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" Oct 30 11:35:44.917048 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.916878 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-swgqc\" (UID: \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" Oct 30 11:35:44.917048 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.916915 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-swgqc\" (UID: \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" Oct 30 11:35:44.917048 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.916938 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74375ab7-73d4-42b5-a149-87e9da66fa59-config\") pod \"data-science-thanos-querier-proxy-7b56895dd5-b4vnq\" (UID: \"74375ab7-73d4-42b5-a149-87e9da66fa59\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" Oct 30 11:35:44.917048 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.916954 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/610b3ae0-322e-4bba-bd8c-22b37b0afd25-client-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq\" (UID: \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" Oct 30 11:35:44.917048 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:35:44.916962 4289 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 30 11:35:44.917048 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:35:44.917037 4289 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-prometheus-client-cert podName:36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf nodeName:}" failed. No retries permitted until 2026-10-30 11:35:45.417016092 +0000 UTC m=+3035.918395574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-swgqc" (UID: "36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf") : secret "prometheus-operated-tls" not found Oct 30 11:35:44.917369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.916971 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc4gp\" (UniqueName: \"kubernetes.io/projected/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-kube-api-access-lc4gp\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-swgqc\" (UID: \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" Oct 30 11:35:44.917369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.917130 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2smwj\" (UniqueName: \"kubernetes.io/projected/74375ab7-73d4-42b5-a149-87e9da66fa59-kube-api-access-2smwj\") pod \"data-science-thanos-querier-proxy-7b56895dd5-b4vnq\" (UID: \"74375ab7-73d4-42b5-a149-87e9da66fa59\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" Oct 30 11:35:44.917369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.917170 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/610b3ae0-322e-4bba-bd8c-22b37b0afd25-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq\" (UID: \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" Oct 30 11:35:44.917369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.917229 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-269fw\" (UniqueName: \"kubernetes.io/projected/5c8a13a5-5433-4a6c-86eb-6447d5831e7b-kube-api-access-269fw\") pod \"korrel8r-5cbcc89b8f-jvktk\" (UID: \"5c8a13a5-5433-4a6c-86eb-6447d5831e7b\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk" Oct 30 11:35:44.917369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.917271 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c8a13a5-5433-4a6c-86eb-6447d5831e7b-config\") pod \"korrel8r-5cbcc89b8f-jvktk\" (UID: \"5c8a13a5-5433-4a6c-86eb-6447d5831e7b\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk" Oct 30 11:35:44.917369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.917300 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5c8a13a5-5433-4a6c-86eb-6447d5831e7b-tls\") pod \"korrel8r-5cbcc89b8f-jvktk\" (UID: \"5c8a13a5-5433-4a6c-86eb-6447d5831e7b\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk" Oct 30 11:35:44.917369 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.917342 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m42hk\" (UniqueName: \"kubernetes.io/projected/610b3ae0-322e-4bba-bd8c-22b37b0afd25-kube-api-access-m42hk\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq\" (UID: \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" Oct 30 11:35:44.917787 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.917376 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/74375ab7-73d4-42b5-a149-87e9da66fa59-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-b4vnq\" (UID: \"74375ab7-73d4-42b5-a149-87e9da66fa59\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" Oct 30 11:35:44.917787 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.917446 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-swgqc\" (UID: \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" Oct 30 11:35:44.917787 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.917487 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/610b3ae0-322e-4bba-bd8c-22b37b0afd25-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq\" (UID: \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" Oct 30 11:35:44.917787 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.917672 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/610b3ae0-322e-4bba-bd8c-22b37b0afd25-client-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq\" (UID: \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" Oct 30 11:35:44.917787 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.917705 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/610b3ae0-322e-4bba-bd8c-22b37b0afd25-config\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq\" (UID: \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" Oct 30 11:35:44.918059 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.917842 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-swgqc\" (UID: \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" Oct 30 11:35:44.918123 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.918095 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/610b3ae0-322e-4bba-bd8c-22b37b0afd25-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq\" (UID: \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" Oct 30 11:35:44.919743 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.919714 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-swgqc\" (UID: \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" Oct 30 11:35:44.919885 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.919779 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-swgqc\" (UID: \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" Oct 30 11:35:44.919885 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.919779 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/610b3ae0-322e-4bba-bd8c-22b37b0afd25-tls\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq\" (UID: \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" Oct 30 11:35:44.940723 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.940695 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m42hk\" (UniqueName: \"kubernetes.io/projected/610b3ae0-322e-4bba-bd8c-22b37b0afd25-kube-api-access-m42hk\") pod \"data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq\" (UID: \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" Oct 30 11:35:44.940856 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.940837 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc4gp\" (UniqueName: \"kubernetes.io/projected/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-kube-api-access-lc4gp\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-swgqc\" (UID: \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" Oct 30 11:35:44.977880 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:44.977850 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:45.017991 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.017949 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74375ab7-73d4-42b5-a149-87e9da66fa59-config\") pod \"data-science-thanos-querier-proxy-7b56895dd5-b4vnq\" (UID: \"74375ab7-73d4-42b5-a149-87e9da66fa59\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" Oct 30 11:35:45.018167 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.018005 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2smwj\" (UniqueName: \"kubernetes.io/projected/74375ab7-73d4-42b5-a149-87e9da66fa59-kube-api-access-2smwj\") pod \"data-science-thanos-querier-proxy-7b56895dd5-b4vnq\" (UID: \"74375ab7-73d4-42b5-a149-87e9da66fa59\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" Oct 30 11:35:45.018167 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.018038 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-269fw\" (UniqueName: \"kubernetes.io/projected/5c8a13a5-5433-4a6c-86eb-6447d5831e7b-kube-api-access-269fw\") pod \"korrel8r-5cbcc89b8f-jvktk\" (UID: \"5c8a13a5-5433-4a6c-86eb-6447d5831e7b\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk" Oct 30 11:35:45.018167 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.018063 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c8a13a5-5433-4a6c-86eb-6447d5831e7b-config\") pod \"korrel8r-5cbcc89b8f-jvktk\" (UID: \"5c8a13a5-5433-4a6c-86eb-6447d5831e7b\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk" Oct 30 11:35:45.018167 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.018086 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5c8a13a5-5433-4a6c-86eb-6447d5831e7b-tls\") pod \"korrel8r-5cbcc89b8f-jvktk\" (UID: \"5c8a13a5-5433-4a6c-86eb-6447d5831e7b\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk" Oct 30 11:35:45.018167 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.018129 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/74375ab7-73d4-42b5-a149-87e9da66fa59-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-b4vnq\" (UID: \"74375ab7-73d4-42b5-a149-87e9da66fa59\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" Oct 30 11:35:45.018782 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.018750 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74375ab7-73d4-42b5-a149-87e9da66fa59-config\") pod \"data-science-thanos-querier-proxy-7b56895dd5-b4vnq\" (UID: \"74375ab7-73d4-42b5-a149-87e9da66fa59\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" Oct 30 11:35:45.018945 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.018925 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c8a13a5-5433-4a6c-86eb-6447d5831e7b-config\") pod \"korrel8r-5cbcc89b8f-jvktk\" (UID: \"5c8a13a5-5433-4a6c-86eb-6447d5831e7b\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk" Oct 30 11:35:45.020366 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.020349 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" Oct 30 11:35:45.020637 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.020617 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/74375ab7-73d4-42b5-a149-87e9da66fa59-tls\") pod \"data-science-thanos-querier-proxy-7b56895dd5-b4vnq\" (UID: \"74375ab7-73d4-42b5-a149-87e9da66fa59\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" Oct 30 11:35:45.020795 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.020776 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5c8a13a5-5433-4a6c-86eb-6447d5831e7b-tls\") pod \"korrel8r-5cbcc89b8f-jvktk\" (UID: \"5c8a13a5-5433-4a6c-86eb-6447d5831e7b\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk" Oct 30 11:35:45.041700 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.041674 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2smwj\" (UniqueName: \"kubernetes.io/projected/74375ab7-73d4-42b5-a149-87e9da66fa59-kube-api-access-2smwj\") pod \"data-science-thanos-querier-proxy-7b56895dd5-b4vnq\" (UID: \"74375ab7-73d4-42b5-a149-87e9da66fa59\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" Oct 30 11:35:45.041816 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.041774 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-269fw\" (UniqueName: \"kubernetes.io/projected/5c8a13a5-5433-4a6c-86eb-6447d5831e7b-kube-api-access-269fw\") pod \"korrel8r-5cbcc89b8f-jvktk\" (UID: \"5c8a13a5-5433-4a6c-86eb-6447d5831e7b\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk" Oct 30 11:35:45.082033 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.081765 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" Oct 30 11:35:45.114995 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.114968 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk" Oct 30 11:35:45.333440 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.333401 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq"] Oct 30 11:35:45.335357 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:35:45.335324 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610b3ae0_322e_4bba_bd8c_22b37b0afd25.slice/crio-768b408166bba05210a5eef9189e105c6ccaf23f415c8dd89bb56575bebf928d WatchSource:0}: Error finding container 768b408166bba05210a5eef9189e105c6ccaf23f415c8dd89bb56575bebf928d: Status 404 returned error can't find the container with id 768b408166bba05210a5eef9189e105c6ccaf23f415c8dd89bb56575bebf928d Oct 30 11:35:45.351166 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.351141 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:45.383048 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.383020 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq"] Oct 30 11:35:45.385418 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:35:45.385385 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74375ab7_73d4_42b5_a149_87e9da66fa59.slice/crio-d2279c4fe2b81ab97d7a972daf336db5d97f62d7e4dbb6dc8bf2abebd2e032df WatchSource:0}: Error finding container d2279c4fe2b81ab97d7a972daf336db5d97f62d7e4dbb6dc8bf2abebd2e032df: Status 404 returned error can't find the container with id d2279c4fe2b81ab97d7a972daf336db5d97f62d7e4dbb6dc8bf2abebd2e032df Oct 30 11:35:45.417730 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.417706 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk"] Oct 30 11:35:45.421085 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.421055 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-swgqc\" (UID: \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" Oct 30 11:35:45.423280 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.423250 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" event={"ID":"610b3ae0-322e-4bba-bd8c-22b37b0afd25","Type":"ContainerStarted","Data":"febbb55b8d1a86a7e23e7cbdff77ed894aeadc8825ce23554434244931178e95"} Oct 30 11:35:45.423375 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.423316 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" event={"ID":"610b3ae0-322e-4bba-bd8c-22b37b0afd25","Type":"ContainerStarted","Data":"768b408166bba05210a5eef9189e105c6ccaf23f415c8dd89bb56575bebf928d"} Oct 30 11:35:45.424253 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.423771 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-swgqc\" (UID: \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" Oct 30 11:35:45.424524 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:35:45.424497 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8a13a5_5433_4a6c_86eb_6447d5831e7b.slice/crio-6d8aed47fced67f95ce0b85b949b10c0635a5032d4976ae41d9f54b426d9ba51 WatchSource:0}: Error finding container 6d8aed47fced67f95ce0b85b949b10c0635a5032d4976ae41d9f54b426d9ba51: Status 404 returned error can't find the container with id 6d8aed47fced67f95ce0b85b949b10c0635a5032d4976ae41d9f54b426d9ba51 Oct 30 11:35:45.424725 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.424691 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" event={"ID":"74375ab7-73d4-42b5-a149-87e9da66fa59","Type":"ContainerStarted","Data":"d2279c4fe2b81ab97d7a972daf336db5d97f62d7e4dbb6dc8bf2abebd2e032df"} Oct 30 11:35:45.532174 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.532148 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:45.574594 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.574565 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" Oct 30 11:35:45.701726 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.701634 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:45.880068 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:45.880042 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc"] Oct 30 11:35:45.882716 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:35:45.882685 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f9b54b_77e9_48d1_8c6d_a3c1f777aaaf.slice/crio-f87e261f171912489ae4f104a766f185f0e2a24132591ab71e09d5617fe03d50 WatchSource:0}: Error finding container f87e261f171912489ae4f104a766f185f0e2a24132591ab71e09d5617fe03d50: Status 404 returned error can't find the container with id f87e261f171912489ae4f104a766f185f0e2a24132591ab71e09d5617fe03d50 Oct 30 11:35:46.299005 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.298975 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:46.431012 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.430975 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk" event={"ID":"5c8a13a5-5433-4a6c-86eb-6447d5831e7b","Type":"ContainerStarted","Data":"df92e74bf13f317dd06ec78ef84fc0bcb910c03e12fe1c9db8a1c7baa6aac02f"} Oct 30 11:35:46.431499 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.431031 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk" event={"ID":"5c8a13a5-5433-4a6c-86eb-6447d5831e7b","Type":"ContainerStarted","Data":"6d8aed47fced67f95ce0b85b949b10c0635a5032d4976ae41d9f54b426d9ba51"} Oct 30 11:35:46.431499 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.431073 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk" Oct 30 11:35:46.433196 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.433168 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" event={"ID":"74375ab7-73d4-42b5-a149-87e9da66fa59","Type":"ContainerStarted","Data":"66ecf4e3b60de14d1f34cec232b5fbebd55d5ea028817ac6eab62124b971c6c9"} Oct 30 11:35:46.433323 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.433224 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" event={"ID":"74375ab7-73d4-42b5-a149-87e9da66fa59","Type":"ContainerStarted","Data":"4562ac08dab70d5b308d6b72c008c0285d80de7d3e367297f8a17eec9f72d7ca"} Oct 30 11:35:46.433323 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.433247 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" event={"ID":"74375ab7-73d4-42b5-a149-87e9da66fa59","Type":"ContainerStarted","Data":"3314719d62551e8dd5ad4ef858093ce2f3ff2765272eea07eeffe9751ea03e4e"} Oct 30 11:35:46.434345 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.434324 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" event={"ID":"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf","Type":"ContainerStarted","Data":"f87e261f171912489ae4f104a766f185f0e2a24132591ab71e09d5617fe03d50"} Oct 30 11:35:46.436030 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.436002 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" event={"ID":"610b3ae0-322e-4bba-bd8c-22b37b0afd25","Type":"ContainerStarted","Data":"5379c7e3371e8603426fec2d4eeb9469361839dfe188875926f9f73ec4b35d27"} Oct 30 11:35:46.436145 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.436044 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" event={"ID":"610b3ae0-322e-4bba-bd8c-22b37b0afd25","Type":"ContainerStarted","Data":"168a5b7ca770bb6bbdd5746a83f79d59e47757ae6c9ec8d165588ac1e7ef25fd"} Oct 30 11:35:46.466739 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.466716 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:46.472834 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.472794 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk" podStartSLOduration=2.472781348 podStartE2EDuration="2.472781348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:46.471500843 +0000 UTC m=+3036.972880330" watchObservedRunningTime="2026-10-30 11:35:46.472781348 +0000 UTC m=+3036.974160833" Oct 30 11:35:46.516115 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.516066 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" podStartSLOduration=2.516053148 podStartE2EDuration="2.516053148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:46.51408231 +0000 UTC m=+3037.015461798" watchObservedRunningTime="2026-10-30 11:35:46.516053148 +0000 UTC m=+3037.017432635" Oct 30 11:35:46.559373 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.559284 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" podStartSLOduration=2.559271251 podStartE2EDuration="2.559271251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:46.557971669 +0000 UTC m=+3037.059351155" watchObservedRunningTime="2026-10-30 11:35:46.559271251 +0000 UTC m=+3037.060650735" Oct 30 11:35:46.638918 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.638884 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:46.817842 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.817783 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:46.942780 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.935737 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-cluster-tls-config\") pod \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " Oct 30 11:35:46.942780 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.935847 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-kube-api-access-khwxs\" (UniqueName: \"kubernetes.io/projected/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-kube-api-access-khwxs\") pod \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " Oct 30 11:35:46.942780 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.935889 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-tls-assets\" (UniqueName: \"kubernetes.io/projected/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-tls-assets\") pod \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " Oct 30 11:35:46.942780 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.935926 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-config-volume\" (UniqueName: \"kubernetes.io/secret/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-config-volume\") pod \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " Oct 30 11:35:46.942780 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.935997 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-config-out\" (UniqueName: \"kubernetes.io/empty-dir/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-config-out\") pod \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " Oct 30 11:35:46.942780 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.936047 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-web-config\" (UniqueName: \"kubernetes.io/secret/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-web-config\") pod \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " Oct 30 11:35:46.942780 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.936092 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-alertmanager-data-science-monitoringstack-db\") pod \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\" (UID: \"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1\") " Oct 30 11:35:46.942780 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.936923 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-alertmanager-data-science-monitoringstack-db" pod "07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1" (UID: "07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:35:46.942780 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.939367 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-config-volume" pod "07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1" (UID: "07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:35:46.942780 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.941593 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-kube-api-access-khwxs" pod "07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1" (UID: "07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1"). InnerVolumeSpecName "kube-api-access-khwxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:35:46.944060 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.943742 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-tls-assets" pod "07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1" (UID: "07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:35:46.945684 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.945658 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-config-out" pod "07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1" (UID: "07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:35:46.949668 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.949645 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-cluster-tls-config" pod "07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1" (UID: "07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:35:46.960375 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:46.960353 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-web-config" pod "07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1" (UID: "07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:35:47.037849 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.037811 4289 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-cluster-tls-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:47.037849 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.037844 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khwxs\" (UniqueName: \"kubernetes.io/projected/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-kube-api-access-khwxs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:47.037849 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.037855 4289 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-tls-assets\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:47.038057 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.037865 4289 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:47.038057 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.037874 4289 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-config-out\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:47.038057 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.037881 4289 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-web-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:47.038057 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.037890 4289 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:35:47.442832 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.442789 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" event={"ID":"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf","Type":"ContainerStarted","Data":"d42782f13851df92b135b24238b5673feeb135f3c87710722ee48522674dcf3a"} Oct 30 11:35:47.443277 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.442894 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" Oct 30 11:35:47.444702 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.444677 4289 generic.go:356] "Generic (PLEG): container finished" podID="07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1" containerID="1e534681b23cba377c3323dcbfb8390e73db95b8be96b633216ba856fbead16a" exitCode=0 Oct 30 11:35:47.444831 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.444747 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 11:35:47.444831 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.444758 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1","Type":"ContainerDied","Data":"1e534681b23cba377c3323dcbfb8390e73db95b8be96b633216ba856fbead16a"} Oct 30 11:35:47.444831 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.444812 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1","Type":"ContainerDied","Data":"8494af37c6aefffd8caeb0400d448b1552cfed4fd26f650271ec2e12293d8337"} Oct 30 11:35:47.444951 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.444842 4289 scope.go:122] "RemoveContainer" containerID="83bddfda9583cdc7934fb3ad3855da07edffc3849260129d131867e78ae7d9a7" Oct 30 11:35:47.454253 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.454237 4289 scope.go:122] "RemoveContainer" containerID="1e534681b23cba377c3323dcbfb8390e73db95b8be96b633216ba856fbead16a" Oct 30 11:35:47.461663 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.461646 4289 scope.go:122] "RemoveContainer" containerID="4199a66f2d10ef1c5f20c661c61199f9315ad17bc11d766a1760d8e0f13ca1e2" Oct 30 11:35:47.468558 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.468538 4289 scope.go:122] "RemoveContainer" containerID="83bddfda9583cdc7934fb3ad3855da07edffc3849260129d131867e78ae7d9a7" Oct 30 11:35:47.468813 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:35:47.468794 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83bddfda9583cdc7934fb3ad3855da07edffc3849260129d131867e78ae7d9a7\": container with ID starting with 83bddfda9583cdc7934fb3ad3855da07edffc3849260129d131867e78ae7d9a7 not found: ID does not exist" containerID="83bddfda9583cdc7934fb3ad3855da07edffc3849260129d131867e78ae7d9a7" Oct 30 11:35:47.468871 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.468820 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83bddfda9583cdc7934fb3ad3855da07edffc3849260129d131867e78ae7d9a7"} err="failed to get container status \"83bddfda9583cdc7934fb3ad3855da07edffc3849260129d131867e78ae7d9a7\": rpc error: code = NotFound desc = could not find container \"83bddfda9583cdc7934fb3ad3855da07edffc3849260129d131867e78ae7d9a7\": container with ID starting with 83bddfda9583cdc7934fb3ad3855da07edffc3849260129d131867e78ae7d9a7 not found: ID does not exist" Oct 30 11:35:47.468871 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.468834 4289 scope.go:122] "RemoveContainer" containerID="1e534681b23cba377c3323dcbfb8390e73db95b8be96b633216ba856fbead16a" Oct 30 11:35:47.469056 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:35:47.469042 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e534681b23cba377c3323dcbfb8390e73db95b8be96b633216ba856fbead16a\": container with ID starting with 1e534681b23cba377c3323dcbfb8390e73db95b8be96b633216ba856fbead16a not found: ID does not exist" containerID="1e534681b23cba377c3323dcbfb8390e73db95b8be96b633216ba856fbead16a" Oct 30 11:35:47.469098 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.469059 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e534681b23cba377c3323dcbfb8390e73db95b8be96b633216ba856fbead16a"} err="failed to get container status \"1e534681b23cba377c3323dcbfb8390e73db95b8be96b633216ba856fbead16a\": rpc error: code = NotFound desc = could not find container \"1e534681b23cba377c3323dcbfb8390e73db95b8be96b633216ba856fbead16a\": container with ID starting with 1e534681b23cba377c3323dcbfb8390e73db95b8be96b633216ba856fbead16a not found: ID does not exist" Oct 30 11:35:47.469098 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.469070 4289 scope.go:122] "RemoveContainer" containerID="4199a66f2d10ef1c5f20c661c61199f9315ad17bc11d766a1760d8e0f13ca1e2" Oct 30 11:35:47.469262 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:35:47.469246 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4199a66f2d10ef1c5f20c661c61199f9315ad17bc11d766a1760d8e0f13ca1e2\": container with ID starting with 4199a66f2d10ef1c5f20c661c61199f9315ad17bc11d766a1760d8e0f13ca1e2 not found: ID does not exist" containerID="4199a66f2d10ef1c5f20c661c61199f9315ad17bc11d766a1760d8e0f13ca1e2" Oct 30 11:35:47.469305 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.469265 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4199a66f2d10ef1c5f20c661c61199f9315ad17bc11d766a1760d8e0f13ca1e2"} err="failed to get container status \"4199a66f2d10ef1c5f20c661c61199f9315ad17bc11d766a1760d8e0f13ca1e2\": rpc error: code = NotFound desc = could not find container \"4199a66f2d10ef1c5f20c661c61199f9315ad17bc11d766a1760d8e0f13ca1e2\": container with ID starting with 4199a66f2d10ef1c5f20c661c61199f9315ad17bc11d766a1760d8e0f13ca1e2 not found: ID does not exist" Oct 30 11:35:47.523235 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.523187 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" podStartSLOduration=3.523175236 podStartE2EDuration="3.523175236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:47.486627804 +0000 UTC m=+3037.988007289" watchObservedRunningTime="2026-10-30 11:35:47.523175236 +0000 UTC m=+3038.024554720" Oct 30 11:35:47.524074 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.524056 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 11:35:47.535746 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.535722 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 11:35:47.624730 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:47.624700 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1" path="/var/lib/kubelet/pods/07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1/volumes" Oct 30 11:35:48.999180 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:48.999142 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:49.304077 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.303973 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:49.467346 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.467313 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:49.695495 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.695457 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:35:49.695892 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.695874 4289 cpu_manager.go:414] "containerMap: removing container" podUID="07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1" containerName="alertmanager" Oct 30 11:35:49.695986 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.695893 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1" containerName="alertmanager" Oct 30 11:35:49.695986 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.695913 4289 cpu_manager.go:414] "containerMap: removing container" podUID="07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1" containerName="init-config-reloader" Oct 30 11:35:49.695986 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.695920 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1" containerName="init-config-reloader" Oct 30 11:35:49.695986 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.695931 4289 cpu_manager.go:414] "containerMap: removing container" podUID="07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1" containerName="config-reloader" Oct 30 11:35:49.695986 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.695939 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1" containerName="config-reloader" Oct 30 11:35:49.696262 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.696065 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1" containerName="alertmanager" Oct 30 11:35:49.696262 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.696080 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ec8e0d-9aa2-4a80-b352-b08b55a9d3e1" containerName="config-reloader" Oct 30 11:35:49.700698 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.700675 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:35:49.725951 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.725912 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:35:49.726365 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.726332 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:49.785449 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.785394 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 11:35:49.785636 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.785502 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 11:35:49.785636 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.785607 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 11:35:49.785636 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.785607 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 11:35:49.785762 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.785653 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-xzf25\"" Oct 30 11:35:49.838853 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.838770 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:35:49.843734 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.843711 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:49.862646 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.862624 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97b3120a-507c-48fa-97b1-ac082a33f83a-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:35:49.862759 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.862659 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97b3120a-507c-48fa-97b1-ac082a33f83a-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:35:49.862759 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.862687 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/97b3120a-507c-48fa-97b1-ac082a33f83a-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:35:49.862862 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.862794 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9djb\" (UniqueName: \"kubernetes.io/projected/97b3120a-507c-48fa-97b1-ac082a33f83a-kube-api-access-m9djb\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:35:49.862862 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.862824 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97b3120a-507c-48fa-97b1-ac082a33f83a-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:35:49.862944 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.862924 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97b3120a-507c-48fa-97b1-ac082a33f83a-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:35:49.862986 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.862947 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97b3120a-507c-48fa-97b1-ac082a33f83a-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:35:49.870322 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.870291 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:35:49.885598 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.885573 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 11:35:49.885728 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.885616 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 11:35:49.885728 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.885665 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 11:35:49.885728 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.885698 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 11:35:49.885880 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.885754 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 11:35:49.885880 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.885810 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-m56vk\"" Oct 30 11:35:49.907582 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.907554 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:49.924514 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.924485 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 11:35:49.924630 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.924587 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 11:35:49.924762 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.924748 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 11:35:49.964182 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.964148 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:49.964337 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.964191 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/97b3120a-507c-48fa-97b1-ac082a33f83a-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:35:49.964337 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.964261 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9djb\" (UniqueName: \"kubernetes.io/projected/97b3120a-507c-48fa-97b1-ac082a33f83a-kube-api-access-m9djb\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:35:49.964337 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.964319 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97b3120a-507c-48fa-97b1-ac082a33f83a-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:35:49.964513 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.964350 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:49.964513 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.964382 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0505cbef-017a-40a4-a07b-426ab937674b-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:49.964594 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.964567 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:49.964663 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.964644 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0505cbef-017a-40a4-a07b-426ab937674b-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:49.964700 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.964651 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/97b3120a-507c-48fa-97b1-ac082a33f83a-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:35:49.964736 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.964706 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dbzg\" (UniqueName: \"kubernetes.io/projected/0505cbef-017a-40a4-a07b-426ab937674b-kube-api-access-7dbzg\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:49.964770 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.964738 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0505cbef-017a-40a4-a07b-426ab937674b-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:49.964807 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.964797 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:49.964841 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.964830 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0505cbef-017a-40a4-a07b-426ab937674b-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:49.964878 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.964858 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97b3120a-507c-48fa-97b1-ac082a33f83a-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:35:49.964928 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.964875 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:49.964928 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.964899 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97b3120a-507c-48fa-97b1-ac082a33f83a-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:35:49.965036 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.964941 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:49.965102 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.965045 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97b3120a-507c-48fa-97b1-ac082a33f83a-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:35:49.965102 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.965085 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0505cbef-017a-40a4-a07b-426ab937674b-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:49.965213 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.965133 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97b3120a-507c-48fa-97b1-ac082a33f83a-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:35:49.967006 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.966977 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97b3120a-507c-48fa-97b1-ac082a33f83a-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:35:49.967150 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.967134 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97b3120a-507c-48fa-97b1-ac082a33f83a-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:35:49.967277 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.967258 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97b3120a-507c-48fa-97b1-ac082a33f83a-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:35:49.967587 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.967567 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97b3120a-507c-48fa-97b1-ac082a33f83a-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:35:49.967832 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.967806 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97b3120a-507c-48fa-97b1-ac082a33f83a-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:35:49.988036 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:49.988010 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9djb\" (UniqueName: \"kubernetes.io/projected/97b3120a-507c-48fa-97b1-ac082a33f83a-kube-api-access-m9djb\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:35:50.019995 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.019968 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:35:50.068716 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.068682 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:50.068931 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.068783 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:50.068931 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.068825 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0505cbef-017a-40a4-a07b-426ab937674b-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:50.068931 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.068875 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:50.068931 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.068901 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0505cbef-017a-40a4-a07b-426ab937674b-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:50.069168 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.068940 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dbzg\" (UniqueName: \"kubernetes.io/projected/0505cbef-017a-40a4-a07b-426ab937674b-kube-api-access-7dbzg\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:50.069168 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.068966 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0505cbef-017a-40a4-a07b-426ab937674b-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:50.069168 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.069000 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:50.069168 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.069035 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0505cbef-017a-40a4-a07b-426ab937674b-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:50.069168 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.069068 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:50.069168 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.069096 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:50.069168 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.069149 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0505cbef-017a-40a4-a07b-426ab937674b-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:50.070143 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.069852 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0505cbef-017a-40a4-a07b-426ab937674b-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:50.070651 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.070620 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0505cbef-017a-40a4-a07b-426ab937674b-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:50.070823 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.070797 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0505cbef-017a-40a4-a07b-426ab937674b-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:50.073377 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.073346 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0505cbef-017a-40a4-a07b-426ab937674b-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:50.073559 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.073530 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:50.074476 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.074405 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:50.074970 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.074854 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0505cbef-017a-40a4-a07b-426ab937674b-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:50.078112 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.075396 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:50.078766 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.078703 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:50.078766 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.078729 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:50.078934 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.078909 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:50.100752 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.100651 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dbzg\" (UniqueName: \"kubernetes.io/projected/0505cbef-017a-40a4-a07b-426ab937674b-kube-api-access-7dbzg\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:50.337276 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.337249 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:35:50.339557 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:35:50.339533 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b3120a_507c_48fa_97b1_ac082a33f83a.slice/crio-20f07ccafd5ddfdd1555074af716d31519699951abb35e5fbdee2f51b99ae380 WatchSource:0}: Error finding container 20f07ccafd5ddfdd1555074af716d31519699951abb35e5fbdee2f51b99ae380: Status 404 returned error can't find the container with id 20f07ccafd5ddfdd1555074af716d31519699951abb35e5fbdee2f51b99ae380 Oct 30 11:35:50.462125 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:50.462091 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"97b3120a-507c-48fa-97b1-ac082a33f83a","Type":"ContainerStarted","Data":"20f07ccafd5ddfdd1555074af716d31519699951abb35e5fbdee2f51b99ae380"} Oct 30 11:35:51.468997 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:51.468960 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"97b3120a-507c-48fa-97b1-ac082a33f83a","Type":"ContainerStarted","Data":"e8ead46ba5c995c4b1cd18200d23de6b0b09ee54afb7b6f7c79e84ff33f24357"} Oct 30 11:35:53.453312 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:53.453278 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" Oct 30 11:35:55.490012 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:55.489974 4289 generic.go:356] "Generic (PLEG): container finished" podID="97b3120a-507c-48fa-97b1-ac082a33f83a" containerID="e8ead46ba5c995c4b1cd18200d23de6b0b09ee54afb7b6f7c79e84ff33f24357" exitCode=0 Oct 30 11:35:55.490417 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:55.490021 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"97b3120a-507c-48fa-97b1-ac082a33f83a","Type":"ContainerDied","Data":"e8ead46ba5c995c4b1cd18200d23de6b0b09ee54afb7b6f7c79e84ff33f24357"} Oct 30 11:35:56.258252 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:35:56.258212 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:35:56.498319 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:56.498290 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"97b3120a-507c-48fa-97b1-ac082a33f83a","Type":"ContainerStarted","Data":"b72c511f69e53eef3fe7bdb225ca7c9a3a3f751df941473436e85dcf73256b10"} Oct 30 11:35:56.498731 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:56.498343 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"97b3120a-507c-48fa-97b1-ac082a33f83a","Type":"ContainerStarted","Data":"fcf8354dc656b6fc44b3421685df836d5bff35411c83cb226a755e52dc230bd0"} Oct 30 11:35:56.498731 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:56.498539 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:35:56.572605 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:56.572550 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.888899609 podStartE2EDuration="7.572536291s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.683636682s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:56.559494773 +0000 UTC m=+3047.060874259" watchObservedRunningTime="2026-10-30 11:35:56.572536291 +0000 UTC m=+3047.073915787" Oct 30 11:35:57.447612 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:57.447576 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk" Oct 30 11:35:58.650858 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:58.650819 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:58.680903 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:58.680873 4289 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:58.751689 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:58.751654 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:58.751873 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:58.751799 4289 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:58.765739 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:58.765711 4289 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") DevicePath \"csi-ea06c66241a0a003a5daff5d9fc602f40e53768b00b104c5e4118b92d558c090\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:58.867591 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:58.867559 4289 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7342da0537aa528cc1545d2018ac3fd0382dfe4f670067e58cf8ba34bb891174/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:58.893953 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:58.893917 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:59.165012 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:59.164977 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:35:59.381731 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:59.381700 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:35:59.495455 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:59.495418 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:35:59.497817 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:35:59.497786 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0505cbef_017a_40a4_a07b_426ab937674b.slice/crio-02e49803d612006073e6da7f1577efae7b73983f5e2fe5ad9345f103a0c6686e WatchSource:0}: Error finding container 02e49803d612006073e6da7f1577efae7b73983f5e2fe5ad9345f103a0c6686e: Status 404 returned error can't find the container with id 02e49803d612006073e6da7f1577efae7b73983f5e2fe5ad9345f103a0c6686e Oct 30 11:35:59.514108 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:35:59.514084 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0505cbef-017a-40a4-a07b-426ab937674b","Type":"ContainerStarted","Data":"02e49803d612006073e6da7f1577efae7b73983f5e2fe5ad9345f103a0c6686e"} Oct 30 11:36:00.520384 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:00.520347 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0505cbef-017a-40a4-a07b-426ab937674b","Type":"ContainerStarted","Data":"ed2d2580f586b86d86ce8037abdc25eb3edf640337d4ad1f798c3b72b2218de6"} Oct 30 11:36:02.505495 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:02.505457 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:36:04.540471 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:04.540414 4289 generic.go:356] "Generic (PLEG): container finished" podID="0505cbef-017a-40a4-a07b-426ab937674b" containerID="ed2d2580f586b86d86ce8037abdc25eb3edf640337d4ad1f798c3b72b2218de6" exitCode=0 Oct 30 11:36:04.540873 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:04.540490 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0505cbef-017a-40a4-a07b-426ab937674b","Type":"ContainerDied","Data":"ed2d2580f586b86d86ce8037abdc25eb3edf640337d4ad1f798c3b72b2218de6"} Oct 30 11:36:05.548048 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:05.548012 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0505cbef-017a-40a4-a07b-426ab937674b","Type":"ContainerStarted","Data":"57f06a3d5151b3503b097bf7eeb49471c1d00c6a088fe4c5dbfe8f61614d4ca1"} Oct 30 11:36:05.548472 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:05.548080 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0505cbef-017a-40a4-a07b-426ab937674b","Type":"ContainerStarted","Data":"8d5c5eb0fac493e20df28125c06bf203232fa67378a1a971b6b1179559fb705a"} Oct 30 11:36:05.548472 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:05.548104 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0505cbef-017a-40a4-a07b-426ab937674b","Type":"ContainerStarted","Data":"8f45374f825a9bd6dcd4b97f414f09b596cc19dab3255132a38e7496c14e9cb9"} Oct 30 11:36:05.628576 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:05.628523 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=12.642956829 podStartE2EDuration="16.628509248s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.985552419s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:36:05.614839947 +0000 UTC m=+3056.116219443" watchObservedRunningTime="2026-10-30 11:36:05.628509248 +0000 UTC m=+3056.129888782" Oct 30 11:36:09.165885 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:09.165855 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:36:14.166063 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:14.166026 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:36:14.171666 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:14.171643 4289 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:36:14.593661 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:14.593632 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:36:15.740563 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:15.740529 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc"] Oct 30 11:36:15.741078 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:15.740716 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" podUID="36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf" containerName="kube-rbac-proxy" containerID="cri-o://d42782f13851df92b135b24238b5673feeb135f3c87710722ee48522674dcf3a" gracePeriod=30 Oct 30 11:36:15.752896 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:15.752872 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq"] Oct 30 11:36:15.753133 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:15.753081 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" podUID="610b3ae0-322e-4bba-bd8c-22b37b0afd25" containerName="kube-rbac-proxy" containerID="cri-o://febbb55b8d1a86a7e23e7cbdff77ed894aeadc8825ce23554434244931178e95" gracePeriod=30 Oct 30 11:36:15.753133 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:15.753094 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" podUID="610b3ae0-322e-4bba-bd8c-22b37b0afd25" containerName="method-gate" containerID="cri-o://168a5b7ca770bb6bbdd5746a83f79d59e47757ae6c9ec8d165588ac1e7ef25fd" gracePeriod=30 Oct 30 11:36:15.753222 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:15.753092 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" podUID="610b3ae0-322e-4bba-bd8c-22b37b0afd25" containerName="prom-label-proxy" containerID="cri-o://5379c7e3371e8603426fec2d4eeb9469361839dfe188875926f9f73ec4b35d27" gracePeriod=30 Oct 30 11:36:15.777850 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:15.777822 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq"] Oct 30 11:36:15.778141 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:15.778118 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" podUID="74375ab7-73d4-42b5-a149-87e9da66fa59" containerName="prom-label-proxy" containerID="cri-o://66ecf4e3b60de14d1f34cec232b5fbebd55d5ea028817ac6eab62124b971c6c9" gracePeriod=30 Oct 30 11:36:15.778206 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:15.778115 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" podUID="74375ab7-73d4-42b5-a149-87e9da66fa59" containerName="kube-rbac-proxy" containerID="cri-o://3314719d62551e8dd5ad4ef858093ce2f3ff2765272eea07eeffe9751ea03e4e" gracePeriod=30 Oct 30 11:36:15.778206 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:15.778116 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" podUID="74375ab7-73d4-42b5-a149-87e9da66fa59" containerName="method-gate" containerID="cri-o://4562ac08dab70d5b308d6b72c008c0285d80de7d3e367297f8a17eec9f72d7ca" gracePeriod=30 Oct 30 11:36:15.794981 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:15.794927 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk"] Oct 30 11:36:15.795138 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:15.795115 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk" podUID="5c8a13a5-5433-4a6c-86eb-6447d5831e7b" containerName="korrel8r" containerID="cri-o://df92e74bf13f317dd06ec78ef84fc0bcb910c03e12fe1c9db8a1c7baa6aac02f" gracePeriod=30 Oct 30 11:36:16.056639 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.056606 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" Oct 30 11:36:16.117119 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.117098 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" Oct 30 11:36:16.125296 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.125267 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk" Oct 30 11:36:16.129918 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.129893 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" Oct 30 11:36:16.218823 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.218777 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/610b3ae0-322e-4bba-bd8c-22b37b0afd25-kube-api-access-m42hk\" (UniqueName: \"kubernetes.io/projected/610b3ae0-322e-4bba-bd8c-22b37b0afd25-kube-api-access-m42hk\") pod \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\" (UID: \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\") " Oct 30 11:36:16.219012 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.218829 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/610b3ae0-322e-4bba-bd8c-22b37b0afd25-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/610b3ae0-322e-4bba-bd8c-22b37b0afd25-prometheus-tls-ca\") pod \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\" (UID: \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\") " Oct 30 11:36:16.219012 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.218867 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c8a13a5-5433-4a6c-86eb-6447d5831e7b-tls\" (UniqueName: \"kubernetes.io/secret/5c8a13a5-5433-4a6c-86eb-6447d5831e7b-tls\") pod \"5c8a13a5-5433-4a6c-86eb-6447d5831e7b\" (UID: \"5c8a13a5-5433-4a6c-86eb-6447d5831e7b\") " Oct 30 11:36:16.219012 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.218902 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c8a13a5-5433-4a6c-86eb-6447d5831e7b-kube-api-access-269fw\" (UniqueName: \"kubernetes.io/projected/5c8a13a5-5433-4a6c-86eb-6447d5831e7b-kube-api-access-269fw\") pod \"5c8a13a5-5433-4a6c-86eb-6447d5831e7b\" (UID: \"5c8a13a5-5433-4a6c-86eb-6447d5831e7b\") " Oct 30 11:36:16.219012 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.218942 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74375ab7-73d4-42b5-a149-87e9da66fa59-kube-api-access-2smwj\" (UniqueName: \"kubernetes.io/projected/74375ab7-73d4-42b5-a149-87e9da66fa59-kube-api-access-2smwj\") pod \"74375ab7-73d4-42b5-a149-87e9da66fa59\" (UID: \"74375ab7-73d4-42b5-a149-87e9da66fa59\") " Oct 30 11:36:16.219012 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.218984 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c8a13a5-5433-4a6c-86eb-6447d5831e7b-config\" (UniqueName: \"kubernetes.io/configmap/5c8a13a5-5433-4a6c-86eb-6447d5831e7b-config\") pod \"5c8a13a5-5433-4a6c-86eb-6447d5831e7b\" (UID: \"5c8a13a5-5433-4a6c-86eb-6447d5831e7b\") " Oct 30 11:36:16.219229 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.219017 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74375ab7-73d4-42b5-a149-87e9da66fa59-config\" (UniqueName: \"kubernetes.io/configmap/74375ab7-73d4-42b5-a149-87e9da66fa59-config\") pod \"74375ab7-73d4-42b5-a149-87e9da66fa59\" (UID: \"74375ab7-73d4-42b5-a149-87e9da66fa59\") " Oct 30 11:36:16.219229 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.219050 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-prometheus-client-cert\") pod \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\" (UID: \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\") " Oct 30 11:36:16.219229 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.219090 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/610b3ae0-322e-4bba-bd8c-22b37b0afd25-tls\" (UniqueName: \"kubernetes.io/secret/610b3ae0-322e-4bba-bd8c-22b37b0afd25-tls\") pod \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\" (UID: \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\") " Oct 30 11:36:16.219229 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.219140 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-prometheus-ca\") pod \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\" (UID: \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\") " Oct 30 11:36:16.219229 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.219193 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-tls\" (UniqueName: \"kubernetes.io/secret/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-tls\") pod \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\" (UID: \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\") " Oct 30 11:36:16.219229 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.219220 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/610b3ae0-322e-4bba-bd8c-22b37b0afd25-client-ca\" (UniqueName: \"kubernetes.io/configmap/610b3ae0-322e-4bba-bd8c-22b37b0afd25-client-ca\") pod \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\" (UID: \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\") " Oct 30 11:36:16.219546 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.219251 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-kube-rbac-proxy-config\") pod \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\" (UID: \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\") " Oct 30 11:36:16.219546 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.219281 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-kube-api-access-lc4gp\" (UniqueName: \"kubernetes.io/projected/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-kube-api-access-lc4gp\") pod \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\" (UID: \"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf\") " Oct 30 11:36:16.219546 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.219316 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/610b3ae0-322e-4bba-bd8c-22b37b0afd25-config\" (UniqueName: \"kubernetes.io/configmap/610b3ae0-322e-4bba-bd8c-22b37b0afd25-config\") pod \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\" (UID: \"610b3ae0-322e-4bba-bd8c-22b37b0afd25\") " Oct 30 11:36:16.219546 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.219356 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74375ab7-73d4-42b5-a149-87e9da66fa59-tls\" (UniqueName: \"kubernetes.io/secret/74375ab7-73d4-42b5-a149-87e9da66fa59-tls\") pod \"74375ab7-73d4-42b5-a149-87e9da66fa59\" (UID: \"74375ab7-73d4-42b5-a149-87e9da66fa59\") " Oct 30 11:36:16.223857 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.223332 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/610b3ae0-322e-4bba-bd8c-22b37b0afd25-config" pod "610b3ae0-322e-4bba-bd8c-22b37b0afd25" (UID: "610b3ae0-322e-4bba-bd8c-22b37b0afd25"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:36:16.223857 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.223374 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-prometheus-ca" pod "36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf" (UID: "36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:36:16.224085 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.223860 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/610b3ae0-322e-4bba-bd8c-22b37b0afd25-prometheus-tls-ca" pod "610b3ae0-322e-4bba-bd8c-22b37b0afd25" (UID: "610b3ae0-322e-4bba-bd8c-22b37b0afd25"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:36:16.224085 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.224055 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c8a13a5-5433-4a6c-86eb-6447d5831e7b-config" pod "5c8a13a5-5433-4a6c-86eb-6447d5831e7b" (UID: "5c8a13a5-5433-4a6c-86eb-6447d5831e7b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:36:16.224263 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.224229 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74375ab7-73d4-42b5-a149-87e9da66fa59-config" pod "74375ab7-73d4-42b5-a149-87e9da66fa59" (UID: "74375ab7-73d4-42b5-a149-87e9da66fa59"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:36:16.224529 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.224501 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/610b3ae0-322e-4bba-bd8c-22b37b0afd25-client-ca" pod "610b3ae0-322e-4bba-bd8c-22b37b0afd25" (UID: "610b3ae0-322e-4bba-bd8c-22b37b0afd25"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:36:16.224635 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.224617 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74375ab7-73d4-42b5-a149-87e9da66fa59-tls" pod "74375ab7-73d4-42b5-a149-87e9da66fa59" (UID: "74375ab7-73d4-42b5-a149-87e9da66fa59"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:36:16.226664 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.226448 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610b3ae0-322e-4bba-bd8c-22b37b0afd25-kube-api-access-m42hk" pod "610b3ae0-322e-4bba-bd8c-22b37b0afd25" (UID: "610b3ae0-322e-4bba-bd8c-22b37b0afd25"). InnerVolumeSpecName "kube-api-access-m42hk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:36:16.227337 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.227313 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-prometheus-client-cert" pod "36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf" (UID: "36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:36:16.227709 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.227670 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-kube-api-access-lc4gp" pod "36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf" (UID: "36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf"). InnerVolumeSpecName "kube-api-access-lc4gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:36:16.227896 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.227873 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-kube-rbac-proxy-config" pod "36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf" (UID: "36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:36:16.228144 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.228122 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-tls" pod "36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf" (UID: "36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:36:16.228230 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.228139 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c8a13a5-5433-4a6c-86eb-6447d5831e7b-kube-api-access-269fw" pod "5c8a13a5-5433-4a6c-86eb-6447d5831e7b" (UID: "5c8a13a5-5433-4a6c-86eb-6447d5831e7b"). InnerVolumeSpecName "kube-api-access-269fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:36:16.228230 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.228169 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/610b3ae0-322e-4bba-bd8c-22b37b0afd25-tls" pod "610b3ae0-322e-4bba-bd8c-22b37b0afd25" (UID: "610b3ae0-322e-4bba-bd8c-22b37b0afd25"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:36:16.228657 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.228636 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74375ab7-73d4-42b5-a149-87e9da66fa59-kube-api-access-2smwj" pod "74375ab7-73d4-42b5-a149-87e9da66fa59" (UID: "74375ab7-73d4-42b5-a149-87e9da66fa59"). InnerVolumeSpecName "kube-api-access-2smwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:36:16.228802 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.228777 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c8a13a5-5433-4a6c-86eb-6447d5831e7b-tls" pod "5c8a13a5-5433-4a6c-86eb-6447d5831e7b" (UID: "5c8a13a5-5433-4a6c-86eb-6447d5831e7b"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:36:16.245348 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.245322 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:36:16.320157 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.320072 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c8a13a5-5433-4a6c-86eb-6447d5831e7b-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:16.320157 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.320098 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74375ab7-73d4-42b5-a149-87e9da66fa59-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:16.320157 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.320107 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-prometheus-client-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:16.320157 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.320118 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/610b3ae0-322e-4bba-bd8c-22b37b0afd25-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:16.320157 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.320128 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-prometheus-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:16.320157 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.320135 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:16.320157 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.320143 4289 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/610b3ae0-322e-4bba-bd8c-22b37b0afd25-client-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:16.320157 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.320150 4289 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-kube-rbac-proxy-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:16.320157 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.320158 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc4gp\" (UniqueName: \"kubernetes.io/projected/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf-kube-api-access-lc4gp\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:16.320157 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.320166 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/610b3ae0-322e-4bba-bd8c-22b37b0afd25-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:16.320832 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.320174 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/74375ab7-73d4-42b5-a149-87e9da66fa59-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:16.320832 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.320182 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m42hk\" (UniqueName: \"kubernetes.io/projected/610b3ae0-322e-4bba-bd8c-22b37b0afd25-kube-api-access-m42hk\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:16.320832 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.320189 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/610b3ae0-322e-4bba-bd8c-22b37b0afd25-prometheus-tls-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:16.320832 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.320197 4289 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5c8a13a5-5433-4a6c-86eb-6447d5831e7b-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:16.320832 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.320205 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-269fw\" (UniqueName: \"kubernetes.io/projected/5c8a13a5-5433-4a6c-86eb-6447d5831e7b-kube-api-access-269fw\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:16.320832 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.320212 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2smwj\" (UniqueName: \"kubernetes.io/projected/74375ab7-73d4-42b5-a149-87e9da66fa59-kube-api-access-2smwj\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:16.331789 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.331759 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:36:16.332072 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.332018 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="97b3120a-507c-48fa-97b1-ac082a33f83a" containerName="alertmanager" containerID="cri-o://fcf8354dc656b6fc44b3421685df836d5bff35411c83cb226a755e52dc230bd0" gracePeriod=120 Oct 30 11:36:16.332072 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.332053 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="97b3120a-507c-48fa-97b1-ac082a33f83a" containerName="config-reloader" containerID="cri-o://b72c511f69e53eef3fe7bdb225ca7c9a3a3f751df941473436e85dcf73256b10" gracePeriod=120 Oct 30 11:36:16.345732 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.345706 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:36:16.449711 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.449675 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:36:16.598662 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.598576 4289 generic.go:356] "Generic (PLEG): container finished" podID="36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf" containerID="d42782f13851df92b135b24238b5673feeb135f3c87710722ee48522674dcf3a" exitCode=0 Oct 30 11:36:16.598662 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.598621 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" Oct 30 11:36:16.598860 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.598654 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" event={"ID":"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf","Type":"ContainerDied","Data":"d42782f13851df92b135b24238b5673feeb135f3c87710722ee48522674dcf3a"} Oct 30 11:36:16.598860 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.598708 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc" event={"ID":"36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf","Type":"ContainerDied","Data":"f87e261f171912489ae4f104a766f185f0e2a24132591ab71e09d5617fe03d50"} Oct 30 11:36:16.598860 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.598739 4289 scope.go:122] "RemoveContainer" containerID="d42782f13851df92b135b24238b5673feeb135f3c87710722ee48522674dcf3a" Oct 30 11:36:16.601051 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.601025 4289 generic.go:356] "Generic (PLEG): container finished" podID="97b3120a-507c-48fa-97b1-ac082a33f83a" containerID="b72c511f69e53eef3fe7bdb225ca7c9a3a3f751df941473436e85dcf73256b10" exitCode=0 Oct 30 11:36:16.601173 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.601095 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"97b3120a-507c-48fa-97b1-ac082a33f83a","Type":"ContainerDied","Data":"b72c511f69e53eef3fe7bdb225ca7c9a3a3f751df941473436e85dcf73256b10"} Oct 30 11:36:16.603108 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.603088 4289 generic.go:356] "Generic (PLEG): container finished" podID="610b3ae0-322e-4bba-bd8c-22b37b0afd25" containerID="5379c7e3371e8603426fec2d4eeb9469361839dfe188875926f9f73ec4b35d27" exitCode=0 Oct 30 11:36:16.603221 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.603109 4289 generic.go:356] "Generic (PLEG): container finished" podID="610b3ae0-322e-4bba-bd8c-22b37b0afd25" containerID="168a5b7ca770bb6bbdd5746a83f79d59e47757ae6c9ec8d165588ac1e7ef25fd" exitCode=0 Oct 30 11:36:16.603221 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.603119 4289 generic.go:356] "Generic (PLEG): container finished" podID="610b3ae0-322e-4bba-bd8c-22b37b0afd25" containerID="febbb55b8d1a86a7e23e7cbdff77ed894aeadc8825ce23554434244931178e95" exitCode=0 Oct 30 11:36:16.603221 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.603157 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" event={"ID":"610b3ae0-322e-4bba-bd8c-22b37b0afd25","Type":"ContainerDied","Data":"5379c7e3371e8603426fec2d4eeb9469361839dfe188875926f9f73ec4b35d27"} Oct 30 11:36:16.603450 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.603220 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" event={"ID":"610b3ae0-322e-4bba-bd8c-22b37b0afd25","Type":"ContainerDied","Data":"168a5b7ca770bb6bbdd5746a83f79d59e47757ae6c9ec8d165588ac1e7ef25fd"} Oct 30 11:36:16.603450 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.603158 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" Oct 30 11:36:16.603450 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.603257 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" event={"ID":"610b3ae0-322e-4bba-bd8c-22b37b0afd25","Type":"ContainerDied","Data":"febbb55b8d1a86a7e23e7cbdff77ed894aeadc8825ce23554434244931178e95"} Oct 30 11:36:16.603450 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.603293 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq" event={"ID":"610b3ae0-322e-4bba-bd8c-22b37b0afd25","Type":"ContainerDied","Data":"768b408166bba05210a5eef9189e105c6ccaf23f415c8dd89bb56575bebf928d"} Oct 30 11:36:16.604756 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.604736 4289 generic.go:356] "Generic (PLEG): container finished" podID="5c8a13a5-5433-4a6c-86eb-6447d5831e7b" containerID="df92e74bf13f317dd06ec78ef84fc0bcb910c03e12fe1c9db8a1c7baa6aac02f" exitCode=0 Oct 30 11:36:16.604878 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.604782 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk" Oct 30 11:36:16.604878 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.604817 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk" event={"ID":"5c8a13a5-5433-4a6c-86eb-6447d5831e7b","Type":"ContainerDied","Data":"df92e74bf13f317dd06ec78ef84fc0bcb910c03e12fe1c9db8a1c7baa6aac02f"} Oct 30 11:36:16.604878 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.604850 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk" event={"ID":"5c8a13a5-5433-4a6c-86eb-6447d5831e7b","Type":"ContainerDied","Data":"6d8aed47fced67f95ce0b85b949b10c0635a5032d4976ae41d9f54b426d9ba51"} Oct 30 11:36:16.607186 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.607163 4289 generic.go:356] "Generic (PLEG): container finished" podID="74375ab7-73d4-42b5-a149-87e9da66fa59" containerID="66ecf4e3b60de14d1f34cec232b5fbebd55d5ea028817ac6eab62124b971c6c9" exitCode=0 Oct 30 11:36:16.607287 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.607189 4289 generic.go:356] "Generic (PLEG): container finished" podID="74375ab7-73d4-42b5-a149-87e9da66fa59" containerID="4562ac08dab70d5b308d6b72c008c0285d80de7d3e367297f8a17eec9f72d7ca" exitCode=0 Oct 30 11:36:16.607287 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.607199 4289 generic.go:356] "Generic (PLEG): container finished" podID="74375ab7-73d4-42b5-a149-87e9da66fa59" containerID="3314719d62551e8dd5ad4ef858093ce2f3ff2765272eea07eeffe9751ea03e4e" exitCode=0 Oct 30 11:36:16.607287 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.607239 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" Oct 30 11:36:16.608159 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.607336 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" event={"ID":"74375ab7-73d4-42b5-a149-87e9da66fa59","Type":"ContainerDied","Data":"66ecf4e3b60de14d1f34cec232b5fbebd55d5ea028817ac6eab62124b971c6c9"} Oct 30 11:36:16.608159 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.607388 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" event={"ID":"74375ab7-73d4-42b5-a149-87e9da66fa59","Type":"ContainerDied","Data":"4562ac08dab70d5b308d6b72c008c0285d80de7d3e367297f8a17eec9f72d7ca"} Oct 30 11:36:16.608159 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.607444 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" event={"ID":"74375ab7-73d4-42b5-a149-87e9da66fa59","Type":"ContainerDied","Data":"3314719d62551e8dd5ad4ef858093ce2f3ff2765272eea07eeffe9751ea03e4e"} Oct 30 11:36:16.608159 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.607484 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq" event={"ID":"74375ab7-73d4-42b5-a149-87e9da66fa59","Type":"ContainerDied","Data":"d2279c4fe2b81ab97d7a972daf336db5d97f62d7e4dbb6dc8bf2abebd2e032df"} Oct 30 11:36:16.608159 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.607644 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="0505cbef-017a-40a4-a07b-426ab937674b" containerName="thanos-sidecar" containerID="cri-o://57f06a3d5151b3503b097bf7eeb49471c1d00c6a088fe4c5dbfe8f61614d4ca1" gracePeriod=600 Oct 30 11:36:16.608159 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.607653 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="0505cbef-017a-40a4-a07b-426ab937674b" containerName="config-reloader" containerID="cri-o://8d5c5eb0fac493e20df28125c06bf203232fa67378a1a971b6b1179559fb705a" gracePeriod=600 Oct 30 11:36:16.608159 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.607581 4289 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="0505cbef-017a-40a4-a07b-426ab937674b" containerName="prometheus" containerID="cri-o://8f45374f825a9bd6dcd4b97f414f09b596cc19dab3255132a38e7496c14e9cb9" gracePeriod=600 Oct 30 11:36:16.616686 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.616669 4289 scope.go:122] "RemoveContainer" containerID="d42782f13851df92b135b24238b5673feeb135f3c87710722ee48522674dcf3a" Oct 30 11:36:16.617495 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:36:16.617472 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42782f13851df92b135b24238b5673feeb135f3c87710722ee48522674dcf3a\": container with ID starting with d42782f13851df92b135b24238b5673feeb135f3c87710722ee48522674dcf3a not found: ID does not exist" containerID="d42782f13851df92b135b24238b5673feeb135f3c87710722ee48522674dcf3a" Oct 30 11:36:16.617657 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.617626 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42782f13851df92b135b24238b5673feeb135f3c87710722ee48522674dcf3a"} err="failed to get container status \"d42782f13851df92b135b24238b5673feeb135f3c87710722ee48522674dcf3a\": rpc error: code = NotFound desc = could not find container \"d42782f13851df92b135b24238b5673feeb135f3c87710722ee48522674dcf3a\": container with ID starting with d42782f13851df92b135b24238b5673feeb135f3c87710722ee48522674dcf3a not found: ID does not exist" Oct 30 11:36:16.617766 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.617659 4289 scope.go:122] "RemoveContainer" containerID="5379c7e3371e8603426fec2d4eeb9469361839dfe188875926f9f73ec4b35d27" Oct 30 11:36:16.628872 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.628850 4289 scope.go:122] "RemoveContainer" containerID="168a5b7ca770bb6bbdd5746a83f79d59e47757ae6c9ec8d165588ac1e7ef25fd" Oct 30 11:36:16.656634 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.656605 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:36:16.728967 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.728943 4289 scope.go:122] "RemoveContainer" containerID="febbb55b8d1a86a7e23e7cbdff77ed894aeadc8825ce23554434244931178e95" Oct 30 11:36:16.737113 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.737096 4289 scope.go:122] "RemoveContainer" containerID="5379c7e3371e8603426fec2d4eeb9469361839dfe188875926f9f73ec4b35d27" Oct 30 11:36:16.737398 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:36:16.737376 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5379c7e3371e8603426fec2d4eeb9469361839dfe188875926f9f73ec4b35d27\": container with ID starting with 5379c7e3371e8603426fec2d4eeb9469361839dfe188875926f9f73ec4b35d27 not found: ID does not exist" containerID="5379c7e3371e8603426fec2d4eeb9469361839dfe188875926f9f73ec4b35d27" Oct 30 11:36:16.737517 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.737410 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5379c7e3371e8603426fec2d4eeb9469361839dfe188875926f9f73ec4b35d27"} err="failed to get container status \"5379c7e3371e8603426fec2d4eeb9469361839dfe188875926f9f73ec4b35d27\": rpc error: code = NotFound desc = could not find container \"5379c7e3371e8603426fec2d4eeb9469361839dfe188875926f9f73ec4b35d27\": container with ID starting with 5379c7e3371e8603426fec2d4eeb9469361839dfe188875926f9f73ec4b35d27 not found: ID does not exist" Oct 30 11:36:16.737517 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.737448 4289 scope.go:122] "RemoveContainer" containerID="168a5b7ca770bb6bbdd5746a83f79d59e47757ae6c9ec8d165588ac1e7ef25fd" Oct 30 11:36:16.737736 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:36:16.737718 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"168a5b7ca770bb6bbdd5746a83f79d59e47757ae6c9ec8d165588ac1e7ef25fd\": container with ID starting with 168a5b7ca770bb6bbdd5746a83f79d59e47757ae6c9ec8d165588ac1e7ef25fd not found: ID does not exist" containerID="168a5b7ca770bb6bbdd5746a83f79d59e47757ae6c9ec8d165588ac1e7ef25fd" Oct 30 11:36:16.737779 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.737742 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"168a5b7ca770bb6bbdd5746a83f79d59e47757ae6c9ec8d165588ac1e7ef25fd"} err="failed to get container status \"168a5b7ca770bb6bbdd5746a83f79d59e47757ae6c9ec8d165588ac1e7ef25fd\": rpc error: code = NotFound desc = could not find container \"168a5b7ca770bb6bbdd5746a83f79d59e47757ae6c9ec8d165588ac1e7ef25fd\": container with ID starting with 168a5b7ca770bb6bbdd5746a83f79d59e47757ae6c9ec8d165588ac1e7ef25fd not found: ID does not exist" Oct 30 11:36:16.737779 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.737758 4289 scope.go:122] "RemoveContainer" containerID="febbb55b8d1a86a7e23e7cbdff77ed894aeadc8825ce23554434244931178e95" Oct 30 11:36:16.737988 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:36:16.737969 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"febbb55b8d1a86a7e23e7cbdff77ed894aeadc8825ce23554434244931178e95\": container with ID starting with febbb55b8d1a86a7e23e7cbdff77ed894aeadc8825ce23554434244931178e95 not found: ID does not exist" containerID="febbb55b8d1a86a7e23e7cbdff77ed894aeadc8825ce23554434244931178e95" Oct 30 11:36:16.738054 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.737996 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"febbb55b8d1a86a7e23e7cbdff77ed894aeadc8825ce23554434244931178e95"} err="failed to get container status \"febbb55b8d1a86a7e23e7cbdff77ed894aeadc8825ce23554434244931178e95\": rpc error: code = NotFound desc = could not find container \"febbb55b8d1a86a7e23e7cbdff77ed894aeadc8825ce23554434244931178e95\": container with ID starting with febbb55b8d1a86a7e23e7cbdff77ed894aeadc8825ce23554434244931178e95 not found: ID does not exist" Oct 30 11:36:16.738054 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.738015 4289 scope.go:122] "RemoveContainer" containerID="5379c7e3371e8603426fec2d4eeb9469361839dfe188875926f9f73ec4b35d27" Oct 30 11:36:16.738256 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.738237 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5379c7e3371e8603426fec2d4eeb9469361839dfe188875926f9f73ec4b35d27"} err="failed to get container status \"5379c7e3371e8603426fec2d4eeb9469361839dfe188875926f9f73ec4b35d27\": rpc error: code = NotFound desc = could not find container \"5379c7e3371e8603426fec2d4eeb9469361839dfe188875926f9f73ec4b35d27\": container with ID starting with 5379c7e3371e8603426fec2d4eeb9469361839dfe188875926f9f73ec4b35d27 not found: ID does not exist" Oct 30 11:36:16.738306 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.738257 4289 scope.go:122] "RemoveContainer" containerID="168a5b7ca770bb6bbdd5746a83f79d59e47757ae6c9ec8d165588ac1e7ef25fd" Oct 30 11:36:16.738538 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.738515 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"168a5b7ca770bb6bbdd5746a83f79d59e47757ae6c9ec8d165588ac1e7ef25fd"} err="failed to get container status \"168a5b7ca770bb6bbdd5746a83f79d59e47757ae6c9ec8d165588ac1e7ef25fd\": rpc error: code = NotFound desc = could not find container \"168a5b7ca770bb6bbdd5746a83f79d59e47757ae6c9ec8d165588ac1e7ef25fd\": container with ID starting with 168a5b7ca770bb6bbdd5746a83f79d59e47757ae6c9ec8d165588ac1e7ef25fd not found: ID does not exist" Oct 30 11:36:16.738623 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.738539 4289 scope.go:122] "RemoveContainer" containerID="febbb55b8d1a86a7e23e7cbdff77ed894aeadc8825ce23554434244931178e95" Oct 30 11:36:16.738780 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.738758 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"febbb55b8d1a86a7e23e7cbdff77ed894aeadc8825ce23554434244931178e95"} err="failed to get container status \"febbb55b8d1a86a7e23e7cbdff77ed894aeadc8825ce23554434244931178e95\": rpc error: code = NotFound desc = could not find container \"febbb55b8d1a86a7e23e7cbdff77ed894aeadc8825ce23554434244931178e95\": container with ID starting with febbb55b8d1a86a7e23e7cbdff77ed894aeadc8825ce23554434244931178e95 not found: ID does not exist" Oct 30 11:36:16.738827 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.738782 4289 scope.go:122] "RemoveContainer" containerID="5379c7e3371e8603426fec2d4eeb9469361839dfe188875926f9f73ec4b35d27" Oct 30 11:36:16.738985 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.738963 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5379c7e3371e8603426fec2d4eeb9469361839dfe188875926f9f73ec4b35d27"} err="failed to get container status \"5379c7e3371e8603426fec2d4eeb9469361839dfe188875926f9f73ec4b35d27\": rpc error: code = NotFound desc = could not find container \"5379c7e3371e8603426fec2d4eeb9469361839dfe188875926f9f73ec4b35d27\": container with ID starting with 5379c7e3371e8603426fec2d4eeb9469361839dfe188875926f9f73ec4b35d27 not found: ID does not exist" Oct 30 11:36:16.739057 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.738985 4289 scope.go:122] "RemoveContainer" containerID="168a5b7ca770bb6bbdd5746a83f79d59e47757ae6c9ec8d165588ac1e7ef25fd" Oct 30 11:36:16.739225 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.739209 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"168a5b7ca770bb6bbdd5746a83f79d59e47757ae6c9ec8d165588ac1e7ef25fd"} err="failed to get container status \"168a5b7ca770bb6bbdd5746a83f79d59e47757ae6c9ec8d165588ac1e7ef25fd\": rpc error: code = NotFound desc = could not find container \"168a5b7ca770bb6bbdd5746a83f79d59e47757ae6c9ec8d165588ac1e7ef25fd\": container with ID starting with 168a5b7ca770bb6bbdd5746a83f79d59e47757ae6c9ec8d165588ac1e7ef25fd not found: ID does not exist" Oct 30 11:36:16.739272 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.739226 4289 scope.go:122] "RemoveContainer" containerID="febbb55b8d1a86a7e23e7cbdff77ed894aeadc8825ce23554434244931178e95" Oct 30 11:36:16.739420 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.739402 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"febbb55b8d1a86a7e23e7cbdff77ed894aeadc8825ce23554434244931178e95"} err="failed to get container status \"febbb55b8d1a86a7e23e7cbdff77ed894aeadc8825ce23554434244931178e95\": rpc error: code = NotFound desc = could not find container \"febbb55b8d1a86a7e23e7cbdff77ed894aeadc8825ce23554434244931178e95\": container with ID starting with febbb55b8d1a86a7e23e7cbdff77ed894aeadc8825ce23554434244931178e95 not found: ID does not exist" Oct 30 11:36:16.739533 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.739422 4289 scope.go:122] "RemoveContainer" containerID="df92e74bf13f317dd06ec78ef84fc0bcb910c03e12fe1c9db8a1c7baa6aac02f" Oct 30 11:36:16.746679 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.746508 4289 scope.go:122] "RemoveContainer" containerID="df92e74bf13f317dd06ec78ef84fc0bcb910c03e12fe1c9db8a1c7baa6aac02f" Oct 30 11:36:16.746894 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:36:16.746767 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df92e74bf13f317dd06ec78ef84fc0bcb910c03e12fe1c9db8a1c7baa6aac02f\": container with ID starting with df92e74bf13f317dd06ec78ef84fc0bcb910c03e12fe1c9db8a1c7baa6aac02f not found: ID does not exist" containerID="df92e74bf13f317dd06ec78ef84fc0bcb910c03e12fe1c9db8a1c7baa6aac02f" Oct 30 11:36:16.746894 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.746786 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df92e74bf13f317dd06ec78ef84fc0bcb910c03e12fe1c9db8a1c7baa6aac02f"} err="failed to get container status \"df92e74bf13f317dd06ec78ef84fc0bcb910c03e12fe1c9db8a1c7baa6aac02f\": rpc error: code = NotFound desc = could not find container \"df92e74bf13f317dd06ec78ef84fc0bcb910c03e12fe1c9db8a1c7baa6aac02f\": container with ID starting with df92e74bf13f317dd06ec78ef84fc0bcb910c03e12fe1c9db8a1c7baa6aac02f not found: ID does not exist" Oct 30 11:36:16.746894 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.746799 4289 scope.go:122] "RemoveContainer" containerID="66ecf4e3b60de14d1f34cec232b5fbebd55d5ea028817ac6eab62124b971c6c9" Oct 30 11:36:16.754151 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.754133 4289 scope.go:122] "RemoveContainer" containerID="4562ac08dab70d5b308d6b72c008c0285d80de7d3e367297f8a17eec9f72d7ca" Oct 30 11:36:16.758788 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.758770 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk"] Oct 30 11:36:16.762726 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.762706 4289 scope.go:122] "RemoveContainer" containerID="3314719d62551e8dd5ad4ef858093ce2f3ff2765272eea07eeffe9751ea03e4e" Oct 30 11:36:16.769507 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.769486 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-jvktk"] Oct 30 11:36:16.783509 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.783489 4289 scope.go:122] "RemoveContainer" containerID="66ecf4e3b60de14d1f34cec232b5fbebd55d5ea028817ac6eab62124b971c6c9" Oct 30 11:36:16.783813 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:36:16.783795 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ecf4e3b60de14d1f34cec232b5fbebd55d5ea028817ac6eab62124b971c6c9\": container with ID starting with 66ecf4e3b60de14d1f34cec232b5fbebd55d5ea028817ac6eab62124b971c6c9 not found: ID does not exist" containerID="66ecf4e3b60de14d1f34cec232b5fbebd55d5ea028817ac6eab62124b971c6c9" Oct 30 11:36:16.783868 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.783825 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ecf4e3b60de14d1f34cec232b5fbebd55d5ea028817ac6eab62124b971c6c9"} err="failed to get container status \"66ecf4e3b60de14d1f34cec232b5fbebd55d5ea028817ac6eab62124b971c6c9\": rpc error: code = NotFound desc = could not find container \"66ecf4e3b60de14d1f34cec232b5fbebd55d5ea028817ac6eab62124b971c6c9\": container with ID starting with 66ecf4e3b60de14d1f34cec232b5fbebd55d5ea028817ac6eab62124b971c6c9 not found: ID does not exist" Oct 30 11:36:16.783868 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.783849 4289 scope.go:122] "RemoveContainer" containerID="4562ac08dab70d5b308d6b72c008c0285d80de7d3e367297f8a17eec9f72d7ca" Oct 30 11:36:16.784094 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:36:16.784078 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4562ac08dab70d5b308d6b72c008c0285d80de7d3e367297f8a17eec9f72d7ca\": container with ID starting with 4562ac08dab70d5b308d6b72c008c0285d80de7d3e367297f8a17eec9f72d7ca not found: ID does not exist" containerID="4562ac08dab70d5b308d6b72c008c0285d80de7d3e367297f8a17eec9f72d7ca" Oct 30 11:36:16.784134 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.784098 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4562ac08dab70d5b308d6b72c008c0285d80de7d3e367297f8a17eec9f72d7ca"} err="failed to get container status \"4562ac08dab70d5b308d6b72c008c0285d80de7d3e367297f8a17eec9f72d7ca\": rpc error: code = NotFound desc = could not find container \"4562ac08dab70d5b308d6b72c008c0285d80de7d3e367297f8a17eec9f72d7ca\": container with ID starting with 4562ac08dab70d5b308d6b72c008c0285d80de7d3e367297f8a17eec9f72d7ca not found: ID does not exist" Oct 30 11:36:16.784134 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.784111 4289 scope.go:122] "RemoveContainer" containerID="3314719d62551e8dd5ad4ef858093ce2f3ff2765272eea07eeffe9751ea03e4e" Oct 30 11:36:16.784370 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:36:16.784353 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3314719d62551e8dd5ad4ef858093ce2f3ff2765272eea07eeffe9751ea03e4e\": container with ID starting with 3314719d62551e8dd5ad4ef858093ce2f3ff2765272eea07eeffe9751ea03e4e not found: ID does not exist" containerID="3314719d62551e8dd5ad4ef858093ce2f3ff2765272eea07eeffe9751ea03e4e" Oct 30 11:36:16.784410 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.784378 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3314719d62551e8dd5ad4ef858093ce2f3ff2765272eea07eeffe9751ea03e4e"} err="failed to get container status \"3314719d62551e8dd5ad4ef858093ce2f3ff2765272eea07eeffe9751ea03e4e\": rpc error: code = NotFound desc = could not find container \"3314719d62551e8dd5ad4ef858093ce2f3ff2765272eea07eeffe9751ea03e4e\": container with ID starting with 3314719d62551e8dd5ad4ef858093ce2f3ff2765272eea07eeffe9751ea03e4e not found: ID does not exist" Oct 30 11:36:16.784410 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.784393 4289 scope.go:122] "RemoveContainer" containerID="66ecf4e3b60de14d1f34cec232b5fbebd55d5ea028817ac6eab62124b971c6c9" Oct 30 11:36:16.784696 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.784674 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ecf4e3b60de14d1f34cec232b5fbebd55d5ea028817ac6eab62124b971c6c9"} err="failed to get container status \"66ecf4e3b60de14d1f34cec232b5fbebd55d5ea028817ac6eab62124b971c6c9\": rpc error: code = NotFound desc = could not find container \"66ecf4e3b60de14d1f34cec232b5fbebd55d5ea028817ac6eab62124b971c6c9\": container with ID starting with 66ecf4e3b60de14d1f34cec232b5fbebd55d5ea028817ac6eab62124b971c6c9 not found: ID does not exist" Oct 30 11:36:16.784760 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.784698 4289 scope.go:122] "RemoveContainer" containerID="4562ac08dab70d5b308d6b72c008c0285d80de7d3e367297f8a17eec9f72d7ca" Oct 30 11:36:16.784982 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.784962 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4562ac08dab70d5b308d6b72c008c0285d80de7d3e367297f8a17eec9f72d7ca"} err="failed to get container status \"4562ac08dab70d5b308d6b72c008c0285d80de7d3e367297f8a17eec9f72d7ca\": rpc error: code = NotFound desc = could not find container \"4562ac08dab70d5b308d6b72c008c0285d80de7d3e367297f8a17eec9f72d7ca\": container with ID starting with 4562ac08dab70d5b308d6b72c008c0285d80de7d3e367297f8a17eec9f72d7ca not found: ID does not exist" Oct 30 11:36:16.785030 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.784982 4289 scope.go:122] "RemoveContainer" containerID="3314719d62551e8dd5ad4ef858093ce2f3ff2765272eea07eeffe9751ea03e4e" Oct 30 11:36:16.785194 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.785177 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3314719d62551e8dd5ad4ef858093ce2f3ff2765272eea07eeffe9751ea03e4e"} err="failed to get container status \"3314719d62551e8dd5ad4ef858093ce2f3ff2765272eea07eeffe9751ea03e4e\": rpc error: code = NotFound desc = could not find container \"3314719d62551e8dd5ad4ef858093ce2f3ff2765272eea07eeffe9751ea03e4e\": container with ID starting with 3314719d62551e8dd5ad4ef858093ce2f3ff2765272eea07eeffe9751ea03e4e not found: ID does not exist" Oct 30 11:36:16.785239 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.785194 4289 scope.go:122] "RemoveContainer" containerID="66ecf4e3b60de14d1f34cec232b5fbebd55d5ea028817ac6eab62124b971c6c9" Oct 30 11:36:16.785417 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.785398 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ecf4e3b60de14d1f34cec232b5fbebd55d5ea028817ac6eab62124b971c6c9"} err="failed to get container status \"66ecf4e3b60de14d1f34cec232b5fbebd55d5ea028817ac6eab62124b971c6c9\": rpc error: code = NotFound desc = could not find container \"66ecf4e3b60de14d1f34cec232b5fbebd55d5ea028817ac6eab62124b971c6c9\": container with ID starting with 66ecf4e3b60de14d1f34cec232b5fbebd55d5ea028817ac6eab62124b971c6c9 not found: ID does not exist" Oct 30 11:36:16.785496 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.785419 4289 scope.go:122] "RemoveContainer" containerID="4562ac08dab70d5b308d6b72c008c0285d80de7d3e367297f8a17eec9f72d7ca" Oct 30 11:36:16.785650 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.785633 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4562ac08dab70d5b308d6b72c008c0285d80de7d3e367297f8a17eec9f72d7ca"} err="failed to get container status \"4562ac08dab70d5b308d6b72c008c0285d80de7d3e367297f8a17eec9f72d7ca\": rpc error: code = NotFound desc = could not find container \"4562ac08dab70d5b308d6b72c008c0285d80de7d3e367297f8a17eec9f72d7ca\": container with ID starting with 4562ac08dab70d5b308d6b72c008c0285d80de7d3e367297f8a17eec9f72d7ca not found: ID does not exist" Oct 30 11:36:16.785650 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.785649 4289 scope.go:122] "RemoveContainer" containerID="3314719d62551e8dd5ad4ef858093ce2f3ff2765272eea07eeffe9751ea03e4e" Oct 30 11:36:16.785900 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.785880 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3314719d62551e8dd5ad4ef858093ce2f3ff2765272eea07eeffe9751ea03e4e"} err="failed to get container status \"3314719d62551e8dd5ad4ef858093ce2f3ff2765272eea07eeffe9751ea03e4e\": rpc error: code = NotFound desc = could not find container \"3314719d62551e8dd5ad4ef858093ce2f3ff2765272eea07eeffe9751ea03e4e\": container with ID starting with 3314719d62551e8dd5ad4ef858093ce2f3ff2765272eea07eeffe9751ea03e4e not found: ID does not exist" Oct 30 11:36:16.805730 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.805704 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq"] Oct 30 11:36:16.810286 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.810253 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-7d9d9fb777-h8csq"] Oct 30 11:36:16.847639 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.847606 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq"] Oct 30 11:36:16.852511 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.852444 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-7b56895dd5-b4vnq"] Oct 30 11:36:16.889187 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.889159 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc"] Oct 30 11:36:16.892121 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:16.892095 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-swgqc"] Oct 30 11:36:17.055901 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.055873 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:36:17.188197 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.188176 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:36:17.298988 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.298953 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:36:17.329709 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.329676 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97b3120a-507c-48fa-97b1-ac082a33f83a-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97b3120a-507c-48fa-97b1-ac082a33f83a-cluster-tls-config\") pod \"97b3120a-507c-48fa-97b1-ac082a33f83a\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " Oct 30 11:36:17.329897 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.329759 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97b3120a-507c-48fa-97b1-ac082a33f83a-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/97b3120a-507c-48fa-97b1-ac082a33f83a-alertmanager-data-science-monitoringstack-db\") pod \"97b3120a-507c-48fa-97b1-ac082a33f83a\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " Oct 30 11:36:17.329897 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.329837 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b3120a-507c-48fa-97b1-ac082a33f83a-tls-assets\" (UniqueName: \"kubernetes.io/projected/97b3120a-507c-48fa-97b1-ac082a33f83a-tls-assets\") pod \"97b3120a-507c-48fa-97b1-ac082a33f83a\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " Oct 30 11:36:17.329897 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.329883 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97b3120a-507c-48fa-97b1-ac082a33f83a-config-out\" (UniqueName: \"kubernetes.io/empty-dir/97b3120a-507c-48fa-97b1-ac082a33f83a-config-out\") pod \"97b3120a-507c-48fa-97b1-ac082a33f83a\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " Oct 30 11:36:17.330071 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.329909 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97b3120a-507c-48fa-97b1-ac082a33f83a-config-volume\" (UniqueName: \"kubernetes.io/secret/97b3120a-507c-48fa-97b1-ac082a33f83a-config-volume\") pod \"97b3120a-507c-48fa-97b1-ac082a33f83a\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " Oct 30 11:36:17.330071 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.329961 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97b3120a-507c-48fa-97b1-ac082a33f83a-web-config\" (UniqueName: \"kubernetes.io/secret/97b3120a-507c-48fa-97b1-ac082a33f83a-web-config\") pod \"97b3120a-507c-48fa-97b1-ac082a33f83a\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " Oct 30 11:36:17.330071 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.329994 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b3120a-507c-48fa-97b1-ac082a33f83a-kube-api-access-m9djb\" (UniqueName: \"kubernetes.io/projected/97b3120a-507c-48fa-97b1-ac082a33f83a-kube-api-access-m9djb\") pod \"97b3120a-507c-48fa-97b1-ac082a33f83a\" (UID: \"97b3120a-507c-48fa-97b1-ac082a33f83a\") " Oct 30 11:36:17.330496 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.330456 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97b3120a-507c-48fa-97b1-ac082a33f83a-alertmanager-data-science-monitoringstack-db" pod "97b3120a-507c-48fa-97b1-ac082a33f83a" (UID: "97b3120a-507c-48fa-97b1-ac082a33f83a"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:36:17.334999 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.334966 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97b3120a-507c-48fa-97b1-ac082a33f83a-config-out" pod "97b3120a-507c-48fa-97b1-ac082a33f83a" (UID: "97b3120a-507c-48fa-97b1-ac082a33f83a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:36:17.340665 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.340628 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b3120a-507c-48fa-97b1-ac082a33f83a-kube-api-access-m9djb" pod "97b3120a-507c-48fa-97b1-ac082a33f83a" (UID: "97b3120a-507c-48fa-97b1-ac082a33f83a"). InnerVolumeSpecName "kube-api-access-m9djb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:36:17.340665 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.340654 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b3120a-507c-48fa-97b1-ac082a33f83a-tls-assets" pod "97b3120a-507c-48fa-97b1-ac082a33f83a" (UID: "97b3120a-507c-48fa-97b1-ac082a33f83a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:36:17.341120 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.341079 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97b3120a-507c-48fa-97b1-ac082a33f83a-config-volume" pod "97b3120a-507c-48fa-97b1-ac082a33f83a" (UID: "97b3120a-507c-48fa-97b1-ac082a33f83a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:36:17.342548 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.342523 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97b3120a-507c-48fa-97b1-ac082a33f83a-cluster-tls-config" pod "97b3120a-507c-48fa-97b1-ac082a33f83a" (UID: "97b3120a-507c-48fa-97b1-ac082a33f83a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:36:17.350381 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.350355 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97b3120a-507c-48fa-97b1-ac082a33f83a-web-config" pod "97b3120a-507c-48fa-97b1-ac082a33f83a" (UID: "97b3120a-507c-48fa-97b1-ac082a33f83a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:36:17.360339 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.360317 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:36:17.431479 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.431385 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0505cbef-017a-40a4-a07b-426ab937674b-kube-api-access-7dbzg\" (UniqueName: \"kubernetes.io/projected/0505cbef-017a-40a4-a07b-426ab937674b-kube-api-access-7dbzg\") pod \"0505cbef-017a-40a4-a07b-426ab937674b\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " Oct 30 11:36:17.431479 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.431471 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"0505cbef-017a-40a4-a07b-426ab937674b\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " Oct 30 11:36:17.431700 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.431500 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0505cbef-017a-40a4-a07b-426ab937674b-tls-assets\" (UniqueName: \"kubernetes.io/projected/0505cbef-017a-40a4-a07b-426ab937674b-tls-assets\") pod \"0505cbef-017a-40a4-a07b-426ab937674b\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " Oct 30 11:36:17.431700 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.431526 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-config\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-config\") pod \"0505cbef-017a-40a4-a07b-426ab937674b\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " Oct 30 11:36:17.431700 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.431555 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0505cbef-017a-40a4-a07b-426ab937674b-config-out\" (UniqueName: \"kubernetes.io/empty-dir/0505cbef-017a-40a4-a07b-426ab937674b-config-out\") pod \"0505cbef-017a-40a4-a07b-426ab937674b\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " Oct 30 11:36:17.431700 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.431585 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0505cbef-017a-40a4-a07b-426ab937674b-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0505cbef-017a-40a4-a07b-426ab937674b-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"0505cbef-017a-40a4-a07b-426ab937674b\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " Oct 30 11:36:17.431700 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.431616 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0505cbef-017a-40a4-a07b-426ab937674b-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0505cbef-017a-40a4-a07b-426ab937674b-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"0505cbef-017a-40a4-a07b-426ab937674b\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " Oct 30 11:36:17.431700 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.431690 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-thanos-prometheus-http-client-file\") pod \"0505cbef-017a-40a4-a07b-426ab937674b\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " Oct 30 11:36:17.432132 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.431744 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"0505cbef-017a-40a4-a07b-426ab937674b\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " Oct 30 11:36:17.432132 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.431787 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-web-config\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-web-config\") pod \"0505cbef-017a-40a4-a07b-426ab937674b\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " Oct 30 11:36:17.432132 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.431821 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0505cbef-017a-40a4-a07b-426ab937674b-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0505cbef-017a-40a4-a07b-426ab937674b-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"0505cbef-017a-40a4-a07b-426ab937674b\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " Oct 30 11:36:17.432132 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.431864 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-secret-prometheus-web-tls-ca\") pod \"0505cbef-017a-40a4-a07b-426ab937674b\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " Oct 30 11:36:17.432132 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.431978 4289 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") pod \"0505cbef-017a-40a4-a07b-426ab937674b\" (UID: \"0505cbef-017a-40a4-a07b-426ab937674b\") " Oct 30 11:36:17.432414 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.432258 4289 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97b3120a-507c-48fa-97b1-ac082a33f83a-web-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:17.432414 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.432278 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9djb\" (UniqueName: \"kubernetes.io/projected/97b3120a-507c-48fa-97b1-ac082a33f83a-kube-api-access-m9djb\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:17.432414 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.432293 4289 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97b3120a-507c-48fa-97b1-ac082a33f83a-cluster-tls-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:17.432414 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.432307 4289 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/97b3120a-507c-48fa-97b1-ac082a33f83a-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:17.432414 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.432322 4289 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97b3120a-507c-48fa-97b1-ac082a33f83a-tls-assets\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:17.432414 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.432336 4289 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97b3120a-507c-48fa-97b1-ac082a33f83a-config-out\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:17.432414 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.432349 4289 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97b3120a-507c-48fa-97b1-ac082a33f83a-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:17.433464 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.433133 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0505cbef-017a-40a4-a07b-426ab937674b-prometheus-data-science-monitoringstack-rulefiles-2" pod "0505cbef-017a-40a4-a07b-426ab937674b" (UID: "0505cbef-017a-40a4-a07b-426ab937674b"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:36:17.433464 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.433384 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0505cbef-017a-40a4-a07b-426ab937674b-prometheus-data-science-monitoringstack-rulefiles-0" pod "0505cbef-017a-40a4-a07b-426ab937674b" (UID: "0505cbef-017a-40a4-a07b-426ab937674b"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:36:17.433834 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.433795 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0505cbef-017a-40a4-a07b-426ab937674b-prometheus-data-science-monitoringstack-rulefiles-1" pod "0505cbef-017a-40a4-a07b-426ab937674b" (UID: "0505cbef-017a-40a4-a07b-426ab937674b"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:36:17.434124 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.434073 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0505cbef-017a-40a4-a07b-426ab937674b-kube-api-access-7dbzg" pod "0505cbef-017a-40a4-a07b-426ab937674b" (UID: "0505cbef-017a-40a4-a07b-426ab937674b"). InnerVolumeSpecName "kube-api-access-7dbzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:36:17.434411 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.434390 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "0505cbef-017a-40a4-a07b-426ab937674b" (UID: "0505cbef-017a-40a4-a07b-426ab937674b"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:36:17.435694 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.435597 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0505cbef-017a-40a4-a07b-426ab937674b-tls-assets" pod "0505cbef-017a-40a4-a07b-426ab937674b" (UID: "0505cbef-017a-40a4-a07b-426ab937674b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:36:17.436122 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.436094 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "0505cbef-017a-40a4-a07b-426ab937674b" (UID: "0505cbef-017a-40a4-a07b-426ab937674b"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:36:17.436287 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.436253 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-config" pod "0505cbef-017a-40a4-a07b-426ab937674b" (UID: "0505cbef-017a-40a4-a07b-426ab937674b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:36:17.436423 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.436398 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-thanos-prometheus-http-client-file" pod "0505cbef-017a-40a4-a07b-426ab937674b" (UID: "0505cbef-017a-40a4-a07b-426ab937674b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:36:17.436904 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.436882 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0505cbef-017a-40a4-a07b-426ab937674b-config-out" pod "0505cbef-017a-40a4-a07b-426ab937674b" (UID: "0505cbef-017a-40a4-a07b-426ab937674b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:36:17.437798 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.437776 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-secret-prometheus-web-tls-ca" pod "0505cbef-017a-40a4-a07b-426ab937674b" (UID: "0505cbef-017a-40a4-a07b-426ab937674b"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:36:17.437968 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.437952 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48" pod "0505cbef-017a-40a4-a07b-426ab937674b" (UID: "0505cbef-017a-40a4-a07b-426ab937674b"). InnerVolumeSpecName "pvc-c799cc48-125a-421c-8fc6-8a4d017a7653". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 11:36:17.448845 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.448818 4289 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-web-config" pod "0505cbef-017a-40a4-a07b-426ab937674b" (UID: "0505cbef-017a-40a4-a07b-426ab937674b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:36:17.516746 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.516717 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:36:17.532742 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.532712 4289 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dbzg\" (UniqueName: \"kubernetes.io/projected/0505cbef-017a-40a4-a07b-426ab937674b-kube-api-access-7dbzg\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:17.532742 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.532738 4289 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:17.532742 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.532749 4289 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0505cbef-017a-40a4-a07b-426ab937674b-tls-assets\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:17.532957 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.532782 4289 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:17.532957 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.532791 4289 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0505cbef-017a-40a4-a07b-426ab937674b-config-out\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:17.532957 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.532800 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0505cbef-017a-40a4-a07b-426ab937674b-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:17.532957 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.532810 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0505cbef-017a-40a4-a07b-426ab937674b-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:17.532957 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.532820 4289 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:17.532957 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.532834 4289 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:17.532957 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.532842 4289 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-web-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:17.532957 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.532854 4289 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0505cbef-017a-40a4-a07b-426ab937674b-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:17.532957 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.532870 4289 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/0505cbef-017a-40a4-a07b-426ab937674b-secret-prometheus-web-tls-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Oct 30 11:36:17.532957 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.532898 4289 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") on node \"ip-10-0-137-242.ec2.internal\" " Oct 30 11:36:17.544612 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.544590 4289 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-c799cc48-125a-421c-8fc6-8a4d017a7653" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48") on node "ip-10-0-137-242.ec2.internal" Oct 30 11:36:17.620794 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.620761 4289 generic.go:356] "Generic (PLEG): container finished" podID="97b3120a-507c-48fa-97b1-ac082a33f83a" containerID="fcf8354dc656b6fc44b3421685df836d5bff35411c83cb226a755e52dc230bd0" exitCode=0 Oct 30 11:36:17.620968 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.620908 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 11:36:17.625836 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.625810 4289 generic.go:356] "Generic (PLEG): container finished" podID="0505cbef-017a-40a4-a07b-426ab937674b" containerID="57f06a3d5151b3503b097bf7eeb49471c1d00c6a088fe4c5dbfe8f61614d4ca1" exitCode=0 Oct 30 11:36:17.625836 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.625830 4289 generic.go:356] "Generic (PLEG): container finished" podID="0505cbef-017a-40a4-a07b-426ab937674b" containerID="8d5c5eb0fac493e20df28125c06bf203232fa67378a1a971b6b1179559fb705a" exitCode=0 Oct 30 11:36:17.625836 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.625839 4289 generic.go:356] "Generic (PLEG): container finished" podID="0505cbef-017a-40a4-a07b-426ab937674b" containerID="8f45374f825a9bd6dcd4b97f414f09b596cc19dab3255132a38e7496c14e9cb9" exitCode=0 Oct 30 11:36:17.626039 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.625924 4289 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 11:36:17.626344 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.626310 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf" path="/var/lib/kubelet/pods/36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf/volumes" Oct 30 11:36:17.627032 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.627011 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c8a13a5-5433-4a6c-86eb-6447d5831e7b" path="/var/lib/kubelet/pods/5c8a13a5-5433-4a6c-86eb-6447d5831e7b/volumes" Oct 30 11:36:17.627540 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.627523 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="610b3ae0-322e-4bba-bd8c-22b37b0afd25" path="/var/lib/kubelet/pods/610b3ae0-322e-4bba-bd8c-22b37b0afd25/volumes" Oct 30 11:36:17.628568 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.628551 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74375ab7-73d4-42b5-a149-87e9da66fa59" path="/var/lib/kubelet/pods/74375ab7-73d4-42b5-a149-87e9da66fa59/volumes" Oct 30 11:36:17.629272 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.629244 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"97b3120a-507c-48fa-97b1-ac082a33f83a","Type":"ContainerDied","Data":"fcf8354dc656b6fc44b3421685df836d5bff35411c83cb226a755e52dc230bd0"} Oct 30 11:36:17.629392 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.629312 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"97b3120a-507c-48fa-97b1-ac082a33f83a","Type":"ContainerDied","Data":"20f07ccafd5ddfdd1555074af716d31519699951abb35e5fbdee2f51b99ae380"} Oct 30 11:36:17.629392 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.629355 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0505cbef-017a-40a4-a07b-426ab937674b","Type":"ContainerDied","Data":"57f06a3d5151b3503b097bf7eeb49471c1d00c6a088fe4c5dbfe8f61614d4ca1"} Oct 30 11:36:17.629518 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.629386 4289 scope.go:122] "RemoveContainer" containerID="b72c511f69e53eef3fe7bdb225ca7c9a3a3f751df941473436e85dcf73256b10" Oct 30 11:36:17.629518 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.629398 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0505cbef-017a-40a4-a07b-426ab937674b","Type":"ContainerDied","Data":"8d5c5eb0fac493e20df28125c06bf203232fa67378a1a971b6b1179559fb705a"} Oct 30 11:36:17.629518 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.629449 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0505cbef-017a-40a4-a07b-426ab937674b","Type":"ContainerDied","Data":"8f45374f825a9bd6dcd4b97f414f09b596cc19dab3255132a38e7496c14e9cb9"} Oct 30 11:36:17.629518 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.629486 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0505cbef-017a-40a4-a07b-426ab937674b","Type":"ContainerDied","Data":"02e49803d612006073e6da7f1577efae7b73983f5e2fe5ad9345f103a0c6686e"} Oct 30 11:36:17.633364 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.633340 4289 reconciler_common.go:299] "Volume detached for volume \"pvc-c799cc48-125a-421c-8fc6-8a4d017a7653\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da8893e7ad2dfa48\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"csi-ea06c66241a0a003a5daff5d9fc602f40e53768b00b104c5e4118b92d558c090\"" Oct 30 11:36:17.639893 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.639875 4289 scope.go:122] "RemoveContainer" containerID="fcf8354dc656b6fc44b3421685df836d5bff35411c83cb226a755e52dc230bd0" Oct 30 11:36:17.647353 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.647332 4289 scope.go:122] "RemoveContainer" containerID="e8ead46ba5c995c4b1cd18200d23de6b0b09ee54afb7b6f7c79e84ff33f24357" Oct 30 11:36:17.654646 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.654629 4289 scope.go:122] "RemoveContainer" containerID="b72c511f69e53eef3fe7bdb225ca7c9a3a3f751df941473436e85dcf73256b10" Oct 30 11:36:17.654897 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:36:17.654879 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b72c511f69e53eef3fe7bdb225ca7c9a3a3f751df941473436e85dcf73256b10\": container with ID starting with b72c511f69e53eef3fe7bdb225ca7c9a3a3f751df941473436e85dcf73256b10 not found: ID does not exist" containerID="b72c511f69e53eef3fe7bdb225ca7c9a3a3f751df941473436e85dcf73256b10" Oct 30 11:36:17.654964 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.654904 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b72c511f69e53eef3fe7bdb225ca7c9a3a3f751df941473436e85dcf73256b10"} err="failed to get container status \"b72c511f69e53eef3fe7bdb225ca7c9a3a3f751df941473436e85dcf73256b10\": rpc error: code = NotFound desc = could not find container \"b72c511f69e53eef3fe7bdb225ca7c9a3a3f751df941473436e85dcf73256b10\": container with ID starting with b72c511f69e53eef3fe7bdb225ca7c9a3a3f751df941473436e85dcf73256b10 not found: ID does not exist" Oct 30 11:36:17.654964 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.654919 4289 scope.go:122] "RemoveContainer" containerID="fcf8354dc656b6fc44b3421685df836d5bff35411c83cb226a755e52dc230bd0" Oct 30 11:36:17.655128 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:36:17.655115 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcf8354dc656b6fc44b3421685df836d5bff35411c83cb226a755e52dc230bd0\": container with ID starting with fcf8354dc656b6fc44b3421685df836d5bff35411c83cb226a755e52dc230bd0 not found: ID does not exist" containerID="fcf8354dc656b6fc44b3421685df836d5bff35411c83cb226a755e52dc230bd0" Oct 30 11:36:17.655170 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.655132 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcf8354dc656b6fc44b3421685df836d5bff35411c83cb226a755e52dc230bd0"} err="failed to get container status \"fcf8354dc656b6fc44b3421685df836d5bff35411c83cb226a755e52dc230bd0\": rpc error: code = NotFound desc = could not find container \"fcf8354dc656b6fc44b3421685df836d5bff35411c83cb226a755e52dc230bd0\": container with ID starting with fcf8354dc656b6fc44b3421685df836d5bff35411c83cb226a755e52dc230bd0 not found: ID does not exist" Oct 30 11:36:17.655170 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.655143 4289 scope.go:122] "RemoveContainer" containerID="e8ead46ba5c995c4b1cd18200d23de6b0b09ee54afb7b6f7c79e84ff33f24357" Oct 30 11:36:17.655693 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:36:17.655405 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8ead46ba5c995c4b1cd18200d23de6b0b09ee54afb7b6f7c79e84ff33f24357\": container with ID starting with e8ead46ba5c995c4b1cd18200d23de6b0b09ee54afb7b6f7c79e84ff33f24357 not found: ID does not exist" containerID="e8ead46ba5c995c4b1cd18200d23de6b0b09ee54afb7b6f7c79e84ff33f24357" Oct 30 11:36:17.655693 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.655452 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ead46ba5c995c4b1cd18200d23de6b0b09ee54afb7b6f7c79e84ff33f24357"} err="failed to get container status \"e8ead46ba5c995c4b1cd18200d23de6b0b09ee54afb7b6f7c79e84ff33f24357\": rpc error: code = NotFound desc = could not find container \"e8ead46ba5c995c4b1cd18200d23de6b0b09ee54afb7b6f7c79e84ff33f24357\": container with ID starting with e8ead46ba5c995c4b1cd18200d23de6b0b09ee54afb7b6f7c79e84ff33f24357 not found: ID does not exist" Oct 30 11:36:17.655693 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.655473 4289 scope.go:122] "RemoveContainer" containerID="57f06a3d5151b3503b097bf7eeb49471c1d00c6a088fe4c5dbfe8f61614d4ca1" Oct 30 11:36:17.662716 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.662698 4289 scope.go:122] "RemoveContainer" containerID="8d5c5eb0fac493e20df28125c06bf203232fa67378a1a971b6b1179559fb705a" Oct 30 11:36:17.670068 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.670051 4289 scope.go:122] "RemoveContainer" containerID="8f45374f825a9bd6dcd4b97f414f09b596cc19dab3255132a38e7496c14e9cb9" Oct 30 11:36:17.673607 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.673586 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:36:17.678769 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.678749 4289 scope.go:122] "RemoveContainer" containerID="ed2d2580f586b86d86ce8037abdc25eb3edf640337d4ad1f798c3b72b2218de6" Oct 30 11:36:17.679847 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.679821 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 11:36:17.686297 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.686280 4289 scope.go:122] "RemoveContainer" containerID="57f06a3d5151b3503b097bf7eeb49471c1d00c6a088fe4c5dbfe8f61614d4ca1" Oct 30 11:36:17.686612 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:36:17.686589 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57f06a3d5151b3503b097bf7eeb49471c1d00c6a088fe4c5dbfe8f61614d4ca1\": container with ID starting with 57f06a3d5151b3503b097bf7eeb49471c1d00c6a088fe4c5dbfe8f61614d4ca1 not found: ID does not exist" containerID="57f06a3d5151b3503b097bf7eeb49471c1d00c6a088fe4c5dbfe8f61614d4ca1" Oct 30 11:36:17.686697 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.686617 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f06a3d5151b3503b097bf7eeb49471c1d00c6a088fe4c5dbfe8f61614d4ca1"} err="failed to get container status \"57f06a3d5151b3503b097bf7eeb49471c1d00c6a088fe4c5dbfe8f61614d4ca1\": rpc error: code = NotFound desc = could not find container \"57f06a3d5151b3503b097bf7eeb49471c1d00c6a088fe4c5dbfe8f61614d4ca1\": container with ID starting with 57f06a3d5151b3503b097bf7eeb49471c1d00c6a088fe4c5dbfe8f61614d4ca1 not found: ID does not exist" Oct 30 11:36:17.686697 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.686635 4289 scope.go:122] "RemoveContainer" containerID="8d5c5eb0fac493e20df28125c06bf203232fa67378a1a971b6b1179559fb705a" Oct 30 11:36:17.686870 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:36:17.686852 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d5c5eb0fac493e20df28125c06bf203232fa67378a1a971b6b1179559fb705a\": container with ID starting with 8d5c5eb0fac493e20df28125c06bf203232fa67378a1a971b6b1179559fb705a not found: ID does not exist" containerID="8d5c5eb0fac493e20df28125c06bf203232fa67378a1a971b6b1179559fb705a" Oct 30 11:36:17.686908 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.686875 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d5c5eb0fac493e20df28125c06bf203232fa67378a1a971b6b1179559fb705a"} err="failed to get container status \"8d5c5eb0fac493e20df28125c06bf203232fa67378a1a971b6b1179559fb705a\": rpc error: code = NotFound desc = could not find container \"8d5c5eb0fac493e20df28125c06bf203232fa67378a1a971b6b1179559fb705a\": container with ID starting with 8d5c5eb0fac493e20df28125c06bf203232fa67378a1a971b6b1179559fb705a not found: ID does not exist" Oct 30 11:36:17.686908 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.686892 4289 scope.go:122] "RemoveContainer" containerID="8f45374f825a9bd6dcd4b97f414f09b596cc19dab3255132a38e7496c14e9cb9" Oct 30 11:36:17.687133 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:36:17.687116 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f45374f825a9bd6dcd4b97f414f09b596cc19dab3255132a38e7496c14e9cb9\": container with ID starting with 8f45374f825a9bd6dcd4b97f414f09b596cc19dab3255132a38e7496c14e9cb9 not found: ID does not exist" containerID="8f45374f825a9bd6dcd4b97f414f09b596cc19dab3255132a38e7496c14e9cb9" Oct 30 11:36:17.687187 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.687136 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f45374f825a9bd6dcd4b97f414f09b596cc19dab3255132a38e7496c14e9cb9"} err="failed to get container status \"8f45374f825a9bd6dcd4b97f414f09b596cc19dab3255132a38e7496c14e9cb9\": rpc error: code = NotFound desc = could not find container \"8f45374f825a9bd6dcd4b97f414f09b596cc19dab3255132a38e7496c14e9cb9\": container with ID starting with 8f45374f825a9bd6dcd4b97f414f09b596cc19dab3255132a38e7496c14e9cb9 not found: ID does not exist" Oct 30 11:36:17.687187 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.687147 4289 scope.go:122] "RemoveContainer" containerID="ed2d2580f586b86d86ce8037abdc25eb3edf640337d4ad1f798c3b72b2218de6" Oct 30 11:36:17.687338 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:36:17.687324 4289 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed2d2580f586b86d86ce8037abdc25eb3edf640337d4ad1f798c3b72b2218de6\": container with ID starting with ed2d2580f586b86d86ce8037abdc25eb3edf640337d4ad1f798c3b72b2218de6 not found: ID does not exist" containerID="ed2d2580f586b86d86ce8037abdc25eb3edf640337d4ad1f798c3b72b2218de6" Oct 30 11:36:17.687376 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.687340 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed2d2580f586b86d86ce8037abdc25eb3edf640337d4ad1f798c3b72b2218de6"} err="failed to get container status \"ed2d2580f586b86d86ce8037abdc25eb3edf640337d4ad1f798c3b72b2218de6\": rpc error: code = NotFound desc = could not find container \"ed2d2580f586b86d86ce8037abdc25eb3edf640337d4ad1f798c3b72b2218de6\": container with ID starting with ed2d2580f586b86d86ce8037abdc25eb3edf640337d4ad1f798c3b72b2218de6 not found: ID does not exist" Oct 30 11:36:17.687376 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.687353 4289 scope.go:122] "RemoveContainer" containerID="57f06a3d5151b3503b097bf7eeb49471c1d00c6a088fe4c5dbfe8f61614d4ca1" Oct 30 11:36:17.687571 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.687553 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f06a3d5151b3503b097bf7eeb49471c1d00c6a088fe4c5dbfe8f61614d4ca1"} err="failed to get container status \"57f06a3d5151b3503b097bf7eeb49471c1d00c6a088fe4c5dbfe8f61614d4ca1\": rpc error: code = NotFound desc = could not find container \"57f06a3d5151b3503b097bf7eeb49471c1d00c6a088fe4c5dbfe8f61614d4ca1\": container with ID starting with 57f06a3d5151b3503b097bf7eeb49471c1d00c6a088fe4c5dbfe8f61614d4ca1 not found: ID does not exist" Oct 30 11:36:17.687611 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.687571 4289 scope.go:122] "RemoveContainer" containerID="8d5c5eb0fac493e20df28125c06bf203232fa67378a1a971b6b1179559fb705a" Oct 30 11:36:17.687753 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.687737 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d5c5eb0fac493e20df28125c06bf203232fa67378a1a971b6b1179559fb705a"} err="failed to get container status \"8d5c5eb0fac493e20df28125c06bf203232fa67378a1a971b6b1179559fb705a\": rpc error: code = NotFound desc = could not find container \"8d5c5eb0fac493e20df28125c06bf203232fa67378a1a971b6b1179559fb705a\": container with ID starting with 8d5c5eb0fac493e20df28125c06bf203232fa67378a1a971b6b1179559fb705a not found: ID does not exist" Oct 30 11:36:17.687753 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.687752 4289 scope.go:122] "RemoveContainer" containerID="8f45374f825a9bd6dcd4b97f414f09b596cc19dab3255132a38e7496c14e9cb9" Oct 30 11:36:17.687968 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.687952 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f45374f825a9bd6dcd4b97f414f09b596cc19dab3255132a38e7496c14e9cb9"} err="failed to get container status \"8f45374f825a9bd6dcd4b97f414f09b596cc19dab3255132a38e7496c14e9cb9\": rpc error: code = NotFound desc = could not find container \"8f45374f825a9bd6dcd4b97f414f09b596cc19dab3255132a38e7496c14e9cb9\": container with ID starting with 8f45374f825a9bd6dcd4b97f414f09b596cc19dab3255132a38e7496c14e9cb9 not found: ID does not exist" Oct 30 11:36:17.688013 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.687967 4289 scope.go:122] "RemoveContainer" containerID="ed2d2580f586b86d86ce8037abdc25eb3edf640337d4ad1f798c3b72b2218de6" Oct 30 11:36:17.688161 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.688147 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed2d2580f586b86d86ce8037abdc25eb3edf640337d4ad1f798c3b72b2218de6"} err="failed to get container status \"ed2d2580f586b86d86ce8037abdc25eb3edf640337d4ad1f798c3b72b2218de6\": rpc error: code = NotFound desc = could not find container \"ed2d2580f586b86d86ce8037abdc25eb3edf640337d4ad1f798c3b72b2218de6\": container with ID starting with ed2d2580f586b86d86ce8037abdc25eb3edf640337d4ad1f798c3b72b2218de6 not found: ID does not exist" Oct 30 11:36:17.688161 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.688160 4289 scope.go:122] "RemoveContainer" containerID="57f06a3d5151b3503b097bf7eeb49471c1d00c6a088fe4c5dbfe8f61614d4ca1" Oct 30 11:36:17.688349 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.688332 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f06a3d5151b3503b097bf7eeb49471c1d00c6a088fe4c5dbfe8f61614d4ca1"} err="failed to get container status \"57f06a3d5151b3503b097bf7eeb49471c1d00c6a088fe4c5dbfe8f61614d4ca1\": rpc error: code = NotFound desc = could not find container \"57f06a3d5151b3503b097bf7eeb49471c1d00c6a088fe4c5dbfe8f61614d4ca1\": container with ID starting with 57f06a3d5151b3503b097bf7eeb49471c1d00c6a088fe4c5dbfe8f61614d4ca1 not found: ID does not exist" Oct 30 11:36:17.688387 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.688350 4289 scope.go:122] "RemoveContainer" containerID="8d5c5eb0fac493e20df28125c06bf203232fa67378a1a971b6b1179559fb705a" Oct 30 11:36:17.688664 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.688648 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d5c5eb0fac493e20df28125c06bf203232fa67378a1a971b6b1179559fb705a"} err="failed to get container status \"8d5c5eb0fac493e20df28125c06bf203232fa67378a1a971b6b1179559fb705a\": rpc error: code = NotFound desc = could not find container \"8d5c5eb0fac493e20df28125c06bf203232fa67378a1a971b6b1179559fb705a\": container with ID starting with 8d5c5eb0fac493e20df28125c06bf203232fa67378a1a971b6b1179559fb705a not found: ID does not exist" Oct 30 11:36:17.688664 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.688663 4289 scope.go:122] "RemoveContainer" containerID="8f45374f825a9bd6dcd4b97f414f09b596cc19dab3255132a38e7496c14e9cb9" Oct 30 11:36:17.688863 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.688847 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f45374f825a9bd6dcd4b97f414f09b596cc19dab3255132a38e7496c14e9cb9"} err="failed to get container status \"8f45374f825a9bd6dcd4b97f414f09b596cc19dab3255132a38e7496c14e9cb9\": rpc error: code = NotFound desc = could not find container \"8f45374f825a9bd6dcd4b97f414f09b596cc19dab3255132a38e7496c14e9cb9\": container with ID starting with 8f45374f825a9bd6dcd4b97f414f09b596cc19dab3255132a38e7496c14e9cb9 not found: ID does not exist" Oct 30 11:36:17.688907 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.688864 4289 scope.go:122] "RemoveContainer" containerID="ed2d2580f586b86d86ce8037abdc25eb3edf640337d4ad1f798c3b72b2218de6" Oct 30 11:36:17.689063 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.689047 4289 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed2d2580f586b86d86ce8037abdc25eb3edf640337d4ad1f798c3b72b2218de6"} err="failed to get container status \"ed2d2580f586b86d86ce8037abdc25eb3edf640337d4ad1f798c3b72b2218de6\": rpc error: code = NotFound desc = could not find container \"ed2d2580f586b86d86ce8037abdc25eb3edf640337d4ad1f798c3b72b2218de6\": container with ID starting with ed2d2580f586b86d86ce8037abdc25eb3edf640337d4ad1f798c3b72b2218de6 not found: ID does not exist" Oct 30 11:36:17.733050 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.733012 4289 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:36:17.735553 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:17.735529 4289 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 11:36:18.054443 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:18.054398 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:36:18.347762 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:18.347663 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:36:19.625281 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:19.625236 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0505cbef-017a-40a4-a07b-426ab937674b" path="/var/lib/kubelet/pods/0505cbef-017a-40a4-a07b-426ab937674b/volumes" Oct 30 11:36:19.625965 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:19.625942 4289 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97b3120a-507c-48fa-97b1-ac082a33f83a" path="/var/lib/kubelet/pods/97b3120a-507c-48fa-97b1-ac082a33f83a/volumes" Oct 30 11:36:19.838157 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:19.838123 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:36:20.047438 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:20.047407 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:36:20.258601 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:20.258568 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:36:26.069111 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:26.069067 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:36:26.256888 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:36:26.256853 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:36:26.278881 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:26.278848 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:36:30.651448 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:30.651409 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-observability-system_odh-observability-54787bff44-25dxm_17e85285-af17-408d-802b-47ae882f8a90/manager/0.log" Oct 30 11:36:32.104257 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:32.104226 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:36:32.281101 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:32.281068 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:36:36.823611 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:36.823581 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vskj5_23abd71b-1521-4aec-9d79-198f0f8e14a2/global-pull-secret-syncer/0.log" Oct 30 11:36:37.417563 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:37.417528 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" Oct 30 11:36:38.017955 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:38.017930 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-242.ec2.internal_6dc9a86d1bbc0350320467df988d567a/haproxy/0.log" Oct 30 11:36:39.244821 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:39.244774 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:36:44.367673 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:44.367639 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5dd764cc5d-b6gfh_c6205124-095e-4ba5-a65b-fe0ee9544087/metrics-server/0.log" Oct 30 11:36:44.497095 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:44.497069 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-jxcjp_efe69eb0-c79d-4ea8-97e6-7bc3af0600cb/monitoring-plugin/0.log" Oct 30 11:36:45.088951 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.088919 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:36:45.111689 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.111656 4289 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx"] Oct 30 11:36:45.112137 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112120 4289 cpu_manager.go:414] "containerMap: removing container" podUID="610b3ae0-322e-4bba-bd8c-22b37b0afd25" containerName="kube-rbac-proxy" Oct 30 11:36:45.112183 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112139 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="610b3ae0-322e-4bba-bd8c-22b37b0afd25" containerName="kube-rbac-proxy" Oct 30 11:36:45.112183 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112155 4289 cpu_manager.go:414] "containerMap: removing container" podUID="74375ab7-73d4-42b5-a149-87e9da66fa59" containerName="prom-label-proxy" Oct 30 11:36:45.112183 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112160 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74375ab7-73d4-42b5-a149-87e9da66fa59" containerName="prom-label-proxy" Oct 30 11:36:45.112183 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112169 4289 cpu_manager.go:414] "containerMap: removing container" podUID="610b3ae0-322e-4bba-bd8c-22b37b0afd25" containerName="method-gate" Oct 30 11:36:45.112183 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112174 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="610b3ae0-322e-4bba-bd8c-22b37b0afd25" containerName="method-gate" Oct 30 11:36:45.112183 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112181 4289 cpu_manager.go:414] "containerMap: removing container" podUID="97b3120a-507c-48fa-97b1-ac082a33f83a" containerName="config-reloader" Oct 30 11:36:45.112183 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112185 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b3120a-507c-48fa-97b1-ac082a33f83a" containerName="config-reloader" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112191 4289 cpu_manager.go:414] "containerMap: removing container" podUID="0505cbef-017a-40a4-a07b-426ab937674b" containerName="config-reloader" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112196 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0505cbef-017a-40a4-a07b-426ab937674b" containerName="config-reloader" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112210 4289 cpu_manager.go:414] "containerMap: removing container" podUID="5c8a13a5-5433-4a6c-86eb-6447d5831e7b" containerName="korrel8r" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112215 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c8a13a5-5433-4a6c-86eb-6447d5831e7b" containerName="korrel8r" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112221 4289 cpu_manager.go:414] "containerMap: removing container" podUID="74375ab7-73d4-42b5-a149-87e9da66fa59" containerName="method-gate" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112226 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74375ab7-73d4-42b5-a149-87e9da66fa59" containerName="method-gate" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112231 4289 cpu_manager.go:414] "containerMap: removing container" podUID="0505cbef-017a-40a4-a07b-426ab937674b" containerName="thanos-sidecar" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112236 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0505cbef-017a-40a4-a07b-426ab937674b" containerName="thanos-sidecar" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112244 4289 cpu_manager.go:414] "containerMap: removing container" podUID="0505cbef-017a-40a4-a07b-426ab937674b" containerName="init-config-reloader" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112248 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0505cbef-017a-40a4-a07b-426ab937674b" containerName="init-config-reloader" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112257 4289 cpu_manager.go:414] "containerMap: removing container" podUID="36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf" containerName="kube-rbac-proxy" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112262 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf" containerName="kube-rbac-proxy" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112270 4289 cpu_manager.go:414] "containerMap: removing container" podUID="0505cbef-017a-40a4-a07b-426ab937674b" containerName="prometheus" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112274 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0505cbef-017a-40a4-a07b-426ab937674b" containerName="prometheus" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112299 4289 cpu_manager.go:414] "containerMap: removing container" podUID="74375ab7-73d4-42b5-a149-87e9da66fa59" containerName="kube-rbac-proxy" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112304 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74375ab7-73d4-42b5-a149-87e9da66fa59" containerName="kube-rbac-proxy" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112317 4289 cpu_manager.go:414] "containerMap: removing container" podUID="610b3ae0-322e-4bba-bd8c-22b37b0afd25" containerName="prom-label-proxy" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112322 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="610b3ae0-322e-4bba-bd8c-22b37b0afd25" containerName="prom-label-proxy" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112328 4289 cpu_manager.go:414] "containerMap: removing container" podUID="97b3120a-507c-48fa-97b1-ac082a33f83a" containerName="init-config-reloader" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112333 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b3120a-507c-48fa-97b1-ac082a33f83a" containerName="init-config-reloader" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112340 4289 cpu_manager.go:414] "containerMap: removing container" podUID="97b3120a-507c-48fa-97b1-ac082a33f83a" containerName="alertmanager" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112344 4289 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b3120a-507c-48fa-97b1-ac082a33f83a" containerName="alertmanager" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112398 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="610b3ae0-322e-4bba-bd8c-22b37b0afd25" containerName="prom-label-proxy" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112404 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="97b3120a-507c-48fa-97b1-ac082a33f83a" containerName="alertmanager" Oct 30 11:36:45.112401 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112411 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="610b3ae0-322e-4bba-bd8c-22b37b0afd25" containerName="kube-rbac-proxy" Oct 30 11:36:45.113167 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112417 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="74375ab7-73d4-42b5-a149-87e9da66fa59" containerName="prom-label-proxy" Oct 30 11:36:45.113167 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112423 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="610b3ae0-322e-4bba-bd8c-22b37b0afd25" containerName="method-gate" Oct 30 11:36:45.113167 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112441 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="97b3120a-507c-48fa-97b1-ac082a33f83a" containerName="config-reloader" Oct 30 11:36:45.113167 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112446 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="0505cbef-017a-40a4-a07b-426ab937674b" containerName="config-reloader" Oct 30 11:36:45.113167 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112453 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c8a13a5-5433-4a6c-86eb-6447d5831e7b" containerName="korrel8r" Oct 30 11:36:45.113167 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112458 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="74375ab7-73d4-42b5-a149-87e9da66fa59" containerName="method-gate" Oct 30 11:36:45.113167 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112463 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="0505cbef-017a-40a4-a07b-426ab937674b" containerName="thanos-sidecar" Oct 30 11:36:45.113167 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112469 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="36f9b54b-77e9-48d1-8c6d-a3c1f777aaaf" containerName="kube-rbac-proxy" Oct 30 11:36:45.113167 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112474 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="0505cbef-017a-40a4-a07b-426ab937674b" containerName="prometheus" Oct 30 11:36:45.113167 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.112479 4289 memory_manager.go:365] "RemoveStaleState removing state" podUID="74375ab7-73d4-42b5-a149-87e9da66fa59" containerName="kube-rbac-proxy" Oct 30 11:36:45.115354 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.115339 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx" Oct 30 11:36:45.130316 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.130279 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tgn5g\"/\"kube-root-ca.crt\"" Oct 30 11:36:45.130316 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.130312 4289 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tgn5g\"/\"openshift-service-ca.crt\"" Oct 30 11:36:45.142067 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.142042 4289 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tgn5g\"/\"default-dockercfg-qskz5\"" Oct 30 11:36:45.143491 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.143467 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx"] Oct 30 11:36:45.167408 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.167379 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/954f92cf-a61e-44ad-9157-8f92e461e7b0-lib-modules\") pod \"perf-node-gather-daemonset-2mfdx\" (UID: \"954f92cf-a61e-44ad-9157-8f92e461e7b0\") " pod="openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx" Oct 30 11:36:45.167576 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.167505 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rmn5\" (UniqueName: \"kubernetes.io/projected/954f92cf-a61e-44ad-9157-8f92e461e7b0-kube-api-access-9rmn5\") pod \"perf-node-gather-daemonset-2mfdx\" (UID: \"954f92cf-a61e-44ad-9157-8f92e461e7b0\") " pod="openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx" Oct 30 11:36:45.167576 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.167555 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/954f92cf-a61e-44ad-9157-8f92e461e7b0-proc\") pod \"perf-node-gather-daemonset-2mfdx\" (UID: \"954f92cf-a61e-44ad-9157-8f92e461e7b0\") " pod="openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx" Oct 30 11:36:45.167677 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.167584 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/954f92cf-a61e-44ad-9157-8f92e461e7b0-sys\") pod \"perf-node-gather-daemonset-2mfdx\" (UID: \"954f92cf-a61e-44ad-9157-8f92e461e7b0\") " pod="openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx" Oct 30 11:36:45.167677 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.167632 4289 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/954f92cf-a61e-44ad-9157-8f92e461e7b0-podres\") pod \"perf-node-gather-daemonset-2mfdx\" (UID: \"954f92cf-a61e-44ad-9157-8f92e461e7b0\") " pod="openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx" Oct 30 11:36:45.265558 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.265524 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:36:45.268481 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.268457 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/954f92cf-a61e-44ad-9157-8f92e461e7b0-sys\") pod \"perf-node-gather-daemonset-2mfdx\" (UID: \"954f92cf-a61e-44ad-9157-8f92e461e7b0\") " pod="openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx" Oct 30 11:36:45.268546 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.268499 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/954f92cf-a61e-44ad-9157-8f92e461e7b0-podres\") pod \"perf-node-gather-daemonset-2mfdx\" (UID: \"954f92cf-a61e-44ad-9157-8f92e461e7b0\") " pod="openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx" Oct 30 11:36:45.268546 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.268522 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/954f92cf-a61e-44ad-9157-8f92e461e7b0-lib-modules\") pod \"perf-node-gather-daemonset-2mfdx\" (UID: \"954f92cf-a61e-44ad-9157-8f92e461e7b0\") " pod="openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx" Oct 30 11:36:45.268623 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.268577 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/954f92cf-a61e-44ad-9157-8f92e461e7b0-sys\") pod \"perf-node-gather-daemonset-2mfdx\" (UID: \"954f92cf-a61e-44ad-9157-8f92e461e7b0\") " pod="openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx" Oct 30 11:36:45.268623 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.268588 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rmn5\" (UniqueName: \"kubernetes.io/projected/954f92cf-a61e-44ad-9157-8f92e461e7b0-kube-api-access-9rmn5\") pod \"perf-node-gather-daemonset-2mfdx\" (UID: \"954f92cf-a61e-44ad-9157-8f92e461e7b0\") " pod="openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx" Oct 30 11:36:45.268697 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.268630 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/954f92cf-a61e-44ad-9157-8f92e461e7b0-lib-modules\") pod \"perf-node-gather-daemonset-2mfdx\" (UID: \"954f92cf-a61e-44ad-9157-8f92e461e7b0\") " pod="openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx" Oct 30 11:36:45.268697 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.268635 4289 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/954f92cf-a61e-44ad-9157-8f92e461e7b0-proc\") pod \"perf-node-gather-daemonset-2mfdx\" (UID: \"954f92cf-a61e-44ad-9157-8f92e461e7b0\") " pod="openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx" Oct 30 11:36:45.268697 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.268582 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/954f92cf-a61e-44ad-9157-8f92e461e7b0-podres\") pod \"perf-node-gather-daemonset-2mfdx\" (UID: \"954f92cf-a61e-44ad-9157-8f92e461e7b0\") " pod="openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx" Oct 30 11:36:45.268697 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.268669 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/954f92cf-a61e-44ad-9157-8f92e461e7b0-proc\") pod \"perf-node-gather-daemonset-2mfdx\" (UID: \"954f92cf-a61e-44ad-9157-8f92e461e7b0\") " pod="openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx" Oct 30 11:36:45.292015 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.291983 4289 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rmn5\" (UniqueName: \"kubernetes.io/projected/954f92cf-a61e-44ad-9157-8f92e461e7b0-kube-api-access-9rmn5\") pod \"perf-node-gather-daemonset-2mfdx\" (UID: \"954f92cf-a61e-44ad-9157-8f92e461e7b0\") " pod="openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx" Oct 30 11:36:45.422284 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.422196 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nbxmg_200291cb-1439-4e54-a820-852ce0fa98ff/node-exporter/0.log" Oct 30 11:36:45.428001 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.427976 4289 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx" Oct 30 11:36:45.547485 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.547454 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nbxmg_200291cb-1439-4e54-a820-852ce0fa98ff/kube-rbac-proxy/0.log" Oct 30 11:36:45.675476 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.675386 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nbxmg_200291cb-1439-4e54-a820-852ce0fa98ff/init-textfile/0.log" Oct 30 11:36:45.723166 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.723135 4289 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx"] Oct 30 11:36:45.726075 ip-10-0-137-242 kubenswrapper[4289]: W1030 11:36:45.726046 4289 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod954f92cf_a61e_44ad_9157_8f92e461e7b0.slice/crio-a970ee0d14ee7308d76daa6570b3d9e8475d6bf46ab8eae2446f5923ae9e55f9 WatchSource:0}: Error finding container a970ee0d14ee7308d76daa6570b3d9e8475d6bf46ab8eae2446f5923ae9e55f9: Status 404 returned error can't find the container with id a970ee0d14ee7308d76daa6570b3d9e8475d6bf46ab8eae2446f5923ae9e55f9 Oct 30 11:36:45.745584 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.745555 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx" event={"ID":"954f92cf-a61e-44ad-9157-8f92e461e7b0","Type":"ContainerStarted","Data":"a970ee0d14ee7308d76daa6570b3d9e8475d6bf46ab8eae2446f5923ae9e55f9"} Oct 30 11:36:45.808866 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.808843 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-z5s49_bab94ec6-608a-4f76-aaa9-54ddffb7d195/kube-rbac-proxy-main/0.log" Oct 30 11:36:45.936186 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:45.936110 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-z5s49_bab94ec6-608a-4f76-aaa9-54ddffb7d195/kube-rbac-proxy-self/0.log" Oct 30 11:36:46.062804 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:46.062770 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-z5s49_bab94ec6-608a-4f76-aaa9-54ddffb7d195/openshift-state-metrics/0.log" Oct 30 11:36:46.750996 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:46.750961 4289 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx" event={"ID":"954f92cf-a61e-44ad-9157-8f92e461e7b0","Type":"ContainerStarted","Data":"d5daa57ea5815040d8db1f0e4c84b3cb2c125e982b777521f46b2ff502192570"} Oct 30 11:36:46.751409 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:46.751035 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx" Oct 30 11:36:46.792154 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:46.792105 4289 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx" podStartSLOduration=1.792090699 podStartE2EDuration="1.792090699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:36:46.791031028 +0000 UTC m=+3097.292410514" watchObservedRunningTime="2026-10-30 11:36:46.792090699 +0000 UTC m=+3097.293470183" Oct 30 11:36:47.359777 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:47.359748 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-xr2hd_6b609664-a123-4c86-9a09-611648959685/prometheus-operator-admission-webhook/0.log" Oct 30 11:36:47.877211 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:47.877178 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c86bc94d6-p9wd8_a9264ef6-62e6-4756-89bc-28496a118fb3/thanos-query/0.log" Oct 30 11:36:48.005288 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:48.005262 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c86bc94d6-p9wd8_a9264ef6-62e6-4756-89bc-28496a118fb3/kube-rbac-proxy-web/0.log" Oct 30 11:36:48.133252 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:48.133173 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c86bc94d6-p9wd8_a9264ef6-62e6-4756-89bc-28496a118fb3/kube-rbac-proxy/0.log" Oct 30 11:36:48.258495 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:48.258466 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c86bc94d6-p9wd8_a9264ef6-62e6-4756-89bc-28496a118fb3/prom-label-proxy/0.log" Oct 30 11:36:48.383880 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:48.383796 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c86bc94d6-p9wd8_a9264ef6-62e6-4756-89bc-28496a118fb3/kube-rbac-proxy-rules/0.log" Oct 30 11:36:48.509283 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:48.509253 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c86bc94d6-p9wd8_a9264ef6-62e6-4756-89bc-28496a118fb3/kube-rbac-proxy-metrics/0.log" Oct 30 11:36:51.831711 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:51.831682 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-747f99888f-6s22s_913c87b9-6c0d-43df-80ab-54ab1a9655de/console/0.log" Oct 30 11:36:51.991703 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:51.991672 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-b69cv_3de9ba8f-15d6-4745-9c27-91e6d5529648/download-server/0.log" Oct 30 11:36:52.765075 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:52.765005 4289 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tgn5g/perf-node-gather-daemonset-2mfdx" Oct 30 11:36:55.184074 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:55.184044 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jbkxt_45be6210-4415-4212-ab2c-dd9638dcd4aa/dns/0.log" Oct 30 11:36:55.317240 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:55.317206 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jbkxt_45be6210-4415-4212-ab2c-dd9638dcd4aa/kube-rbac-proxy/0.log" Oct 30 11:36:56.134780 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:56.134751 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gll67_c64ab741-bc4f-4006-932a-214cf53b6dc3/dns-node-resolver/0.log" Oct 30 11:36:56.257766 ip-10-0-137-242 kubenswrapper[4289]: E1030 11:36:56.257734 4289 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cn9sm" containerName="extractor" Oct 30 11:36:57.378461 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:57.378412 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-678d79cf9b-nm8bj_d2541bab-5182-4b00-86ca-6d80327f2d51/registry/0.log" Oct 30 11:36:57.792552 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:36:57.792524 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tvrzn_7437f570-02f5-4131-8b0b-140ed029a668/node-ca/0.log" Oct 30 11:37:00.549198 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:37:00.549161 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gffz9_cbeb8de3-2377-41cf-8fcd-5cb832935d89/serve-healthcheck-canary/0.log" Oct 30 11:37:01.921354 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:37:01.921324 4289 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xmhg_fa6114a1-4640-4ede-86a2-d6bb4aa4e2d9/konnectivity-agent/0.log" limit=1024 Oct 30 11:37:02.114741 ip-10-0-137-242 kubenswrapper[4289]: I1030 11:37:02.114708 4289 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cn9sm_6b7b910a-9274-41cb-8d26-d4a4b269bfea/kube-rbac-proxy/0.log"